Oct 30 19:27:16.006913 ip-10-0-138-61 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 19:27:16.006923 ip-10-0-138-61 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 19:27:16.006933 ip-10-0-138-61 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 19:27:16.007258 ip-10-0-138-61 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 19:27:25.876014 ip-10-0-138-61 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Oct 30 19:27:25.876026 ip-10-0-138-61 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b5a1ddeded9a45c29f476c6a545fdd1d --
Oct 30 19:29:17.784210 ip-10-0-138-61 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:29:18.233404 ip-10-0-138-61 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:29:18.233404 ip-10-0-138-61 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:29:18.233404 ip-10-0-138-61 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:29:18.233404 ip-10-0-138-61 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:29:18.236694 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236598    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:29:18.236694 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236692    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:29:18.236694 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236696    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:29:18.236694 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236699    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:29:18.236694 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236702    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236706    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236709    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236712    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236715    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236718    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236720    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236723    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236726    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236728    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236731    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236733    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236737    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236739    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236742    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236744    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236747    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236750    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236752    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236755    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:29:18.236919 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236757    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236760    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236762    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236765    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236778    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236781    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236784    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236787    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236789    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236792    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236795    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236797    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236800    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236803    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236806    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236809    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236811    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236814    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236817    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:29:18.237612 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236819    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236822    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236825    2569 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236827    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236830    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236832    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236835    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236837    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236840    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236842    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236845    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236848    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236851    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236853    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236856    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236874    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236878    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236882    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236887    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236891    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:29:18.238287 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236895    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236900    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236903    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236905    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236908    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236912    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236915    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236918    2569 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236921    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236924    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236928    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236930    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236933    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236937    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236939    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236944    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236949    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236952    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236956    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:29:18.239018 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236958    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236962    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236966    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236970    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236972    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236975    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236978    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236981    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236985    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236989    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236995    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.236998    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.237001    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.237005    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.237008    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.237011    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.237013    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.237016    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.237019    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:29:18.239692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.237021    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.237024    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.237027    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.237030    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.237033    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238434    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238445    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238449    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238452    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238455    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238458    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238461    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238463    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238466    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238469    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238472    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238474    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238477    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238479    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:29:18.240379 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238482    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238484    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238487    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238489    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238493    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238496    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238498    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238502    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238504    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238507    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238510    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238512    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238515    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238518    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238521    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238523    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238525    2569 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238528    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238530    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238533    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:29:18.241065 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238535    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238538    2569 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238540    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238543    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238546    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238548    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238551    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238554    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238557    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238559    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238562    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238565    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238567    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238570    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238573    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238576    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238578    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238585    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238588    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238590    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:29:18.241774 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238596    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238599    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238602    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238606    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238610    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238614    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238617    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238619    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238622    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238625    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238628    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238631    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238633    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238636    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238639    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238643    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238645    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238648    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238651    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:29:18.242610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238653    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238656    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238658    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238661    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238663    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238666    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238668    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238671    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238673    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238676    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238680    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238682    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238685    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238688    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238691    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238694    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238697    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238699    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238702    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238705    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:29:18.243302 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238709    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238712    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238715    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238718    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238721    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238724    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238727    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238729    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238733    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238736    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238739    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238741    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.238744    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239401    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239409    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239412    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239415    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239418    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239420    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:29:18.244029 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239423    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239428    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239432    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239437    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239440    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239443    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239446    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239449    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239452    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239455    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239458    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239461    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239464    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239466    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239469    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239472    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239474    2569 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239477    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239480    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:29:18.244714 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239482    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239485    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239487    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239490    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239492    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239495    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239498    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239501    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239503    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239506    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239508    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239511    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239513    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239516    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239518    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239521    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239524    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239527    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239529    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239532    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:29:18.245382 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239534    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239537    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239540    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239542    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239548    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239551    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239554    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239557    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239560    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239562    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239565    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239568    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239571    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239574    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239576    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239579    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239581    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239584    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239586    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239589    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:29:18.246124 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239591    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239594    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239597    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239600    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239602    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239604    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239607    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239610    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239613    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239615    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239618    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239620    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239623    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239625    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239628    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239631    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239635    2569 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239639    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239642    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239644    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:29:18.246873 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239647    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239650    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239652    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239655    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239657    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239660    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239662    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239665    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239668    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239670    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239675    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239679    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239682    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239684    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239687    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239690    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239692    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239695    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239698    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:29:18.247584 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239700    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.239703    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240439    2569 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240450    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240458    2569 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240463    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240468    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240471    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240476    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240482    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240486    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240489    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240493    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240496    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240499    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240503    2569 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240506    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240509    2569 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240512    2569 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240515    2569 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240519    2569 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240522    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240525    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240528    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:29:18.248310 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240532    2569 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240536    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240539    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240542    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240545    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240548    2569 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240551    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240554    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240558    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240562    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240567    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240571    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240574    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240577    2569 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240580    2569 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240584    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240589    2569 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240594    2569 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240597    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240600    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240604    2569 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240608    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240611    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240615    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240618    2569 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:29:18.249209 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240621    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240624    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240627    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240630    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240633    2569 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240636    2569 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240639    2569 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240643    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240646    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240650    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240653    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240656    2569 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240660    2569 flags.go:64] FLAG: --help="false"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240663    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240666    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240670    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240674    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240678    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240681    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240684    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240688    2569 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240690    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240693    2569 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:29:18.250164 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240698    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240701    2569 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240705    2569 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240708    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240711    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240714    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240717    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240720    2569 flags.go:64] FLAG: --lock-file=""
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240723    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240726    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240729    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240735    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240738    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240741    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240744    2569 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240747    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240751    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240754    2569 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240757    2569 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240762    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240765    2569 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240770    2569 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240773    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240776    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240779    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:29:18.251060 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240784    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240787    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240790    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240793    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240801    2569 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240804    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240808    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240811    2569 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240814    2569 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240817    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240820    2569 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240824    2569 flags.go:64] FLAG: --port="10250"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240827    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240830    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-088203cde8f433334"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240833    2569 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240836    2569 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240839    2569 flags.go:64] FLAG: --register-node="true"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240842    2569 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240846    2569 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240849    2569 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240852    2569 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240855    2569 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240875    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240878    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240881    2569 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240885    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:29:18.251965 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240888    2569 flags.go:64] FLAG: --runonce="false"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240891    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240894    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240897    2569 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240900    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240903    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240908    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240911    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240915    2569 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240918    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240923    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240926    2569 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240929    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240932    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240935    2569 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240938    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240946    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240949    2569 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240952    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240957    2569 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240960    2569 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240963    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240966    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240969    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240972    2569 flags.go:64] FLAG: --v="2"
Oct 30 19:29:18.252911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240977    2569 flags.go:64] FLAG: --version="false"
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240981    2569 flags.go:64] FLAG: --vmodule=""
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240985    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.240989    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241163    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241171    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241177    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241182    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241187    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241191    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241196    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241200    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241204    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241209    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241212    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241214    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241217    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241222    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241224    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241229    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:29:18.253765 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241232    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241235    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241238    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241241    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241243    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241247    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241250    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241252    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241255    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241258    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241260    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241263    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241265    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241268    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241270    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241273    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241276    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241278    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241281    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241284    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:29:18.254487 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241286    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241289    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241291    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241294    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241296    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241300    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241303    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241305    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241308    2569 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241312    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241317    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241320    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241323    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241326    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241330    2569 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241333    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241336    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241339    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241345    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:29:18.255243 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241347    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241351    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241353    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241356    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241359    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241361    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241364    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241366    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241369    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241371    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241374    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241376    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241379    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241382    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241384    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241387    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241389    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241392    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241396    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241399    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:29:18.256011 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241402    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241405    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241410    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241412    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241415    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241418    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241420    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241423    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241426    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241428    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241431    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241433    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241437    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241439    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241442    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241444    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241447    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241449    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241452    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241454    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:29:18.256710 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241457    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:29:18.257424 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241459    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:29:18.257424 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241462    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:29:18.257424 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241465    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:29:18.257424 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241467    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:29:18.257424 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241471    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:29:18.257424 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241473    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:29:18.257424 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241476    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:29:18.257424 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241478    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:29:18.257424 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241481    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:29:18.257424 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241485    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:29:18.257424 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.241517    2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:29:18.257424 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241559    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:29:18.257424 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241564    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:29:18.257424 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241567    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:29:18.257424 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241572    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:29:18.257424 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241574    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241577    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241580    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241582    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241585    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241588    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241590    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241593    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241596    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241599    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241602    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241605    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241607    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241610    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241613    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241616    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241618    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241621    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241623    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241626    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:29:18.257995 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241629    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241631    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241634    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241636    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241639    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241642    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241646    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241649    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241651    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241655    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241659    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241663    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241666    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241668    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241671    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241673    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241676    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241678    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241681    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:29:18.258692 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241684    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241686    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241689    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241692    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241694    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241697    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241699    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241702    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241705    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241707    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241710    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241712    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241715    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241717    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241720    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241722    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241725    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241728    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241731    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241735    2569 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:29:18.259408 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241738    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241742    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241746    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241749    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241754    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241757    2569 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241760    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241762    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241765    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241767    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241770    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241772    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241775    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241778    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241781    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241783    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241786    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241789    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241792    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:29:18.260127 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241794    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241797    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241800    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241802    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241805    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241807    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241810    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241813    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241815    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241818    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241821    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241823    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241827    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241830    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241832    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241835    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241838    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241843    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241845    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241848    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:29:18.260805 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241850    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241853    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241855    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.241873    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.249201    2569 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.249222    2569 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.249234    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.249239    2569 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249304    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249311    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249314    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249317    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249319    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249322    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249324    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249327    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249331    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249335    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:29:18.261525 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249339    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249342    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249344    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249347    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249349    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249352    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249355    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249357    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249360    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249363    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249365    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249368    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249370    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249373    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249375    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249378    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249380    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249383    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249386    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:29:18.262174 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249396    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249399    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249401    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249404    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249407    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249409    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249412    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249415    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249417    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249420    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249422    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249425    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249428    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249431    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249433    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249436    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249438    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249441    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249443    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249446    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:29:18.262905 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249448    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249451    2569 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249453    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249456    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249459    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249461    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249464    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249466    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249469    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249471    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249475    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249479    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249483    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249492    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249495    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249505    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249508    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249510    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249513    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:29:18.263649 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249515    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249518    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249520    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249523    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249526    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249528    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249531    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249533    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249536    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249538    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249541    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249543    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249546    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249549    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249552    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249554    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249557    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249560    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249563    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249565    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:29:18.264415 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249568    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249571    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249573    2569 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249576    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249579    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249583    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249585    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249596    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249599    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249602    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249604    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249607    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249610    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249612    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249615    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249617    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249620    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:29:18.265140 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249622    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:29:18.265741 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.249654    2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:29:18.265741 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249847    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:29:18.265741 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249851    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:29:18.265741 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249855    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:29:18.265741 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249873    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:29:18.265741 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249877    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:29:18.265741 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249880    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:29:18.265741 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249882    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:29:18.265741 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249885    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:29:18.265741 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249888    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:29:18.265741 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249891    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:29:18.265741 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249893    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:29:18.265741 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249896    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:29:18.265741 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249898    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:29:18.265741 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249901    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249904    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249906    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249909    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249912    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249915    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249918    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249920    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249933    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249936    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249939    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249942    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249944    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249947    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249949    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249952    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249954    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249957    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249959    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:29:18.266284 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249963    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249966    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249969    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249972    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249974    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249977    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249979    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249982    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249984    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249987    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249990    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249992    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249995    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.249997    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250000    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250002    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250005    2569 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250007    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250010    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250013    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:29:18.267033 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250016    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250018    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250029    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250031    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250034    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250037    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250039    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250042    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250045    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250047    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250050    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250053    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250055    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250057    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250060    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250063    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250065    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250068    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250070    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250073    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:29:18.267605 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250075    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250077    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250080    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250082    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250085    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250087    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250090    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250092    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250095    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250098    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250100    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250103    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250105    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250108    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250112    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250121    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250124    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250127    2569 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250129    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250132    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:29:18.268177 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250134    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:29:18.268772 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250137    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:29:18.268772 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250139    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:29:18.268772 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250142    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:29:18.268772 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250144    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:29:18.268772 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250147    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:29:18.268772 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250150    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:29:18.268772 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250152    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:29:18.268772 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250155    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:29:18.268772 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250157    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:29:18.268772 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250160    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:29:18.268772 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250162    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:29:18.268772 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250165    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:29:18.268772 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250168    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:29:18.268772 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.250193    2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:29:18.268772 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250232    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:29:18.268772 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250236    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250239    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250242    2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250245    2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250248    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250251    2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250253    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250256    2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250258    2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250261    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250263    2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250268    2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250276    2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250279    2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250283    2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250287    2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250290    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250292    2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250295    2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:29:18.269205 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250297    2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250300    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250303    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250306    2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250309    2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250311    2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250314    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250316    2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250319    2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250322    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250325    2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250327    2569 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250330    2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250332    2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250335    2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250338    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250341    2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250344    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250347    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:29:18.269736 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250349    2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250352    2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250354    2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250356    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250359    2569 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250363    2569 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250366    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250374    2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250377    2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250379    2569 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250382    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250384    2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250387    2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250390    2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250392    2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250395    2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250397    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250400    2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250402    2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250405    2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:29:18.270254 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250407    2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250410    2569 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250412    2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250417    2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250420    2569 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250422    2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250425    2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250427    2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250430    2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250432    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250435    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250438    2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250440    2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250443    2569 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250446    2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250454    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250457    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250461    2569 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250463    2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250466    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:29:18.270727 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250474    2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250477    2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250479    2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250482    2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250485    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250487    2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250490    2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250492    2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250495    2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250497    2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250500    2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250502    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250506    2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250510    2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250513    2569 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250517    2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250520    2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250523    2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250526    2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:29:18.271292 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250528    2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:29:18.271850 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250531    2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:29:18.271850 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250534    2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:29:18.271850 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250536    2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:29:18.271850 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250539    2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:29:18.271850 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250541    2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:29:18.271850 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250544    2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:29:18.271850 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:18.250547    2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:29:18.271850 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.250624    2569 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:29:18.271850 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.255943    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 19:29:18.271850 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.257014    2569 server.go:1014] "Starting client certificate rotation"
Oct 30 19:29:18.271850 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.257117    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:29:18.271850 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.257142    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:29:18.271850 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.259634    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:29:18.271850 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.260986    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:29:18.271850 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.271207    2569 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:29:18.275269 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.275254    2569 log.go:25] "Validated CRI v1 image API"
Oct 30 19:29:18.276165 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.276142    2569 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:29:18.283566 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.283531    2569 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8d1b0409-7b13-43bc-a7f3-6bc62e421630:/dev/nvme0n1p4 cd23f126-2bbd-49ef-98c2-6d8fd187918c:/dev/nvme0n1p3]
Oct 30 19:29:18.283566 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.283562    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 19:29:18.290873 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.290727    2569 manager.go:217] Machine: {Timestamp:2026-10-30 19:29:18.288095061 +0000 UTC m=+0.372221469 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3073384 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2901f6ef5fcc215b05c36c65585c58 SystemUUID:ec2901f6-ef5f-cc21-5b05-c36c65585c58 BootID:b5a1dded-ed9a-45c2-9f47-6c6a545fdd1d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d3:54:57:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d3:54:57:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:bb:15:d6:9b:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:29:18.290873 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.290850    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:29:18.291019 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.291004    2569 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:29:18.294416 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.294386    2569 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:29:18.294615 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.294417    2569 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-61.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:29:18.294667 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.294618    2569 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:29:18.294667 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.294627    2569 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:29:18.294667 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.294644    2569 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:29:18.294667 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.294659    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:29:18.295525 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.295514    2569 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:29:18.296107 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.296087    2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:29:18.298959 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.298946    2569 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:29:18.299080 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.299070    2569 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:29:18.299112 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.299106    2569 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:29:18.299139 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.299116    2569 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:29:18.299139 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.299126    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:29:18.299139 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.299137    2569 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:29:18.299230 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.299145    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:29:18.299518 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.299507    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:29:18.303284 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.303255    2569 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:29:18.309297 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.309280    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:29:18.310207 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.310194    2569 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:29:18.310254 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.310231    2569 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:29:18.310413 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.310402    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:29:18.310448 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.310418    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:29:18.310448 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.310424    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:29:18.310448 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.310431    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:29:18.310448 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.310436    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:29:18.310448 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.310442    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:29:18.311145 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.311136    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:29:18.311145 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.311146    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:29:18.311199 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.311153    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:29:18.311199 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.311159    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:29:18.311199 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.311167    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:29:18.311199 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.311178    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:29:18.311901 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.311892    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:29:18.311901 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.311902    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:29:18.314540 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.314502    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-61.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 19:29:18.316090 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.316078    2569 server.go:1263] "Started kubelet"
Oct 30 19:29:18.316300 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.316236    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:29:18.316370 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.316324    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:29:18.316370 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.316248    2569 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:29:18.317073 ip-10-0-138-61 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:29:18.318179 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.318149    2569 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-61.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 19:29:18.318323 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.318152    2569 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 19:29:18.318323 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.318317    2569 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:29:18.321811 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.321787    2569 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:29:18.323924 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.323905    2569 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 19:29:18.326391 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.326374    2569 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:29:18.326472 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.326401    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:29:18.327231 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.326520    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:18.327231 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.327036    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:29:18.327231 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.327046    2569 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:29:18.327231 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.327063    2569 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:29:18.327231 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.327176    2569 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:29:18.327231 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.327186    2569 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:29:18.327569 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.327338    2569 factory.go:55] Registering systemd factory
Oct 30 19:29:18.327569 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.327353    2569 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:29:18.327569 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.327563    2569 factory.go:153] Registering CRI-O factory
Oct 30 19:29:18.327749 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.327576    2569 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:29:18.327749 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.327658    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:29:18.327749 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.327677    2569 factory.go:103] Registering Raw factory
Oct 30 19:29:18.327749 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.327692    2569 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:29:18.328482 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.328468    2569 manager.go:319] Starting recovery of all containers
Oct 30 19:29:18.333259 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.333209    2569 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-61.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 19:29:18.333659 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.333630    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:29:18.335843 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.335828    2569 manager.go:324] Recovery completed
Oct 30 19:29:18.339602 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.333313    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-61.us-east-2.compute.internal.18e365645aad71e0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-61.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-61.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:29:18.316040672 +0000 UTC m=+0.400167078,LastTimestamp:2026-10-30 19:29:18.316040672 +0000 UTC m=+0.400167078,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-61.us-east-2.compute.internal,}"
Oct 30 19:29:18.342904 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.341455    2569 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 19:29:18.346691 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.346670    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lj282"
Oct 30 19:29:18.349880 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.349850    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:29:18.352018 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.351999    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:29:18.352113 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.352036    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:29:18.352113 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.352052    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:29:18.352590 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.352574    2569 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:29:18.352590 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.352589    2569 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:29:18.352691 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.352607    2569 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:29:18.352691 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.352636    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:29:18.353400 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.353383    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:29:18.353455 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.353415    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:29:18.353455 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.353429    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:29:18.354040 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.354027    2569 policy_none.go:50] "Start"
Oct 30 19:29:18.354078 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.354045    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:29:18.354078 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.354058    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:29:18.354871 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.354775    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-61.us-east-2.compute.internal.18e365645cd26c0b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-61.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-61.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-61.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:29:18.352018443 +0000 UTC m=+0.436144858,LastTimestamp:2026-10-30 19:29:18.352018443 +0000 UTC m=+0.436144858,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-61.us-east-2.compute.internal,}"
Oct 30 19:29:18.355160 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.355146    2569 policy_none.go:44] "Start"
Oct 30 19:29:18.363135 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.363113    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lj282"
Oct 30 19:29:18.393549 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.393518    2569 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:29:18.395962 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.395941    2569 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:29:18.396080 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.396053    2569 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:29:18.396080 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.396066    2569 server.go:87] "Starting device plugin registration server"
Oct 30 19:29:18.396367 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.396353    2569 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:29:18.396403 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.396369    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:29:18.396489 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.396474    2569 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:29:18.396577 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.396560    2569 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:29:18.396577 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.396572    2569 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:29:18.397089 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.397066    2569 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:29:18.397160 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.397118    2569 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:18.474640 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.474608    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:29:18.475735 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.475722    2569 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:29:18.475807 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.475768    2569 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:29:18.475858 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.475814    2569 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 19:29:18.480772 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.480749    2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:29:18.496991 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.496899    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:29:18.497889 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.497858    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:29:18.497949 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.497905    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:29:18.497949 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.497916    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:29:18.497949 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.497941    2569 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.509055 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.509030    2569 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.509055 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.509057    2569 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-61.us-east-2.compute.internal\": node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:18.557361 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.557325    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:18.576413 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.576387    2569 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal"]
Oct 30 19:29:18.576485 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.576456    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:29:18.578480 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.578462    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:29:18.578599 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.578496    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:29:18.578599 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.578510    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:29:18.579821 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.579805    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:29:18.579986 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.579970    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.580038 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.580005    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:29:18.580692 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.580673    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:29:18.580791 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.580701    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:29:18.580791 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.580711    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:29:18.580791 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.580672    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:29:18.580791 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.580776    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:29:18.580791 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.580788    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:29:18.581920 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.581905    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.581973 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.581938    2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:29:18.582659 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.582643    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:29:18.582758 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.582667    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:29:18.582758 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.582682    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:29:18.609995 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.609963    2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found" node="ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.613930 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.613910    2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found" node="ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.628594 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.628563    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/caaa86be00fc4b40c2a164ffe289a741-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"caaa86be00fc4b40c2a164ffe289a741\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.628687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.628597    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caaa86be00fc4b40c2a164ffe289a741-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"caaa86be00fc4b40c2a164ffe289a741\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.628687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.628616    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8f0f705095219e55fd87172f433b51c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"f8f0f705095219e55fd87172f433b51c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.657821 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.657636    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:18.729073 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.729030    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/caaa86be00fc4b40c2a164ffe289a741-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"caaa86be00fc4b40c2a164ffe289a741\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.729073 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.729076    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caaa86be00fc4b40c2a164ffe289a741-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"caaa86be00fc4b40c2a164ffe289a741\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.729222 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.729096    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8f0f705095219e55fd87172f433b51c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"f8f0f705095219e55fd87172f433b51c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.729222 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.729167    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/caaa86be00fc4b40c2a164ffe289a741-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"caaa86be00fc4b40c2a164ffe289a741\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.729222 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.729174    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caaa86be00fc4b40c2a164ffe289a741-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"caaa86be00fc4b40c2a164ffe289a741\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.729222 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.729169    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8f0f705095219e55fd87172f433b51c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"f8f0f705095219e55fd87172f433b51c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.758209 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.758150    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:18.859806 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.859757    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:18.914081 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.914041    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.916951 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:18.916929    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:18.960490 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:18.960449    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:19.063017 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:19.062921    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:19.165674 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:19.165635    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:19.256634 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:19.256601    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 19:29:19.257212 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:19.256805    2569 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:29:19.266524 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:19.266490    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:19.367678 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:19.367597    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:19.368826 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:19.368784    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:24:18 +0000 UTC" deadline="2028-06-12 07:15:06.711504232 +0000 UTC"
Oct 30 19:29:19.368902 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:19.368827    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14171h45m47.342682305s"
Oct 30 19:29:19.390531 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:19.390495    2569 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:29:19.468029 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:19.467992    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:19.562479 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:19.562434    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaaa86be00fc4b40c2a164ffe289a741.slice/crio-a7a16614b766a578de04bc5f01c9de025ad846dc68802f7bae453eef46f936e8 WatchSource:0}: Error finding container a7a16614b766a578de04bc5f01c9de025ad846dc68802f7bae453eef46f936e8: Status 404 returned error can't find the container with id a7a16614b766a578de04bc5f01c9de025ad846dc68802f7bae453eef46f936e8
Oct 30 19:29:19.562979 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:19.562953    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f0f705095219e55fd87172f433b51c.slice/crio-5f1402c6a45b330bbb1cc8cd1e011283fcbb6dbc621e3c8679a903e3fa69bbd0 WatchSource:0}: Error finding container 5f1402c6a45b330bbb1cc8cd1e011283fcbb6dbc621e3c8679a903e3fa69bbd0: Status 404 returned error can't find the container with id 5f1402c6a45b330bbb1cc8cd1e011283fcbb6dbc621e3c8679a903e3fa69bbd0
Oct 30 19:29:19.568609 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:19.568575    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:19.568609 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:19.568603    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:29:19.671160 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:19.671074    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:19.682279 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:19.682243    2569 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:29:19.771796 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:19.771756    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:19.872792 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:19.872745    2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:29:19.883126 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:19.883100    2569 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:29:19.927483 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:19.927404    2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:19.943004 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:19.942980    2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:29:19.944148 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:19.944132    2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:29:19.951778 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:19.951758    2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:29:20.302244 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.302162    2569 apiserver.go:52] "Watching apiserver"
Oct 30 19:29:20.308631 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.308605    2569 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:29:20.309497 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.309464    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-ncbz9","openshift-dns/node-resolver-2sdcp","openshift-multus/multus-additional-cni-plugins-wg8f2","kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-d4k5h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7","openshift-cluster-node-tuning-operator/tuned-fcz7q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-6wwsm","openshift-image-registry/node-ca-lcxk5","openshift-multus/multus-k6rd4","openshift-multus/network-metrics-daemon-87b4p","openshift-network-operator/iptables-alerter-f2s9d","kube-system/global-pull-secret-syncer-4zdqg"]
Oct 30 19:29:20.311982 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.311955    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:29:20.314532 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.314510    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:29:20.315694 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.315667    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:29:20.316075 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.315849    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:29:20.316197 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.316156    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:29:20.318213 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.315934    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xs4jl\""
Oct 30 19:29:20.318213 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.317473    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:29:20.318213 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.317534    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:29:20.318213 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.317929    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:29:20.318213 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.318150    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zqbbj\""
Oct 30 19:29:20.318539 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.318491    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.321126 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.321107    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:20.321216 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:20.321186    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zdqg" podUID="bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e"
Oct 30 19:29:20.322050 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.321652    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:29:20.322050 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.321673    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:29:20.322050 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.321688    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:29:20.322050 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.321723    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:29:20.322050 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.321853    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:29:20.322050 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.321894    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wp4kx\""
Oct 30 19:29:20.323559 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.323538    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.326137 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.326114    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.327671 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.327650    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 19:29:20.328381 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.328362    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:29:20.328527 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.328505    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:20.328647 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:20.328626    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wwsm" podUID="926e8fa4-1a22-48b1-832e-ea924ef61178"
Oct 30 19:29:20.331083 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.331060    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.333875 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.333850    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:29:20.336465 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.336440    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:29:20.338266 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338241    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovnkube-script-lib\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.338390 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338280    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22dvx\" (UniqueName: \"kubernetes.io/projected/c6ec634f-81f4-4d4e-97ef-33092b38052a-kube-api-access-22dvx\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.338390 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338339    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-systemd-units\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.338578 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338397    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-system-cni-dir\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.338578 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338430    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.338578 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338500    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da310e19-8c58-4785-a88e-09edc8a63e2e-serviceca\") pod \"node-ca-lcxk5\" (UID: \"da310e19-8c58-4785-a88e-09edc8a63e2e\") " pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:29:20.338578 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338533    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcfz2\" (UniqueName: \"kubernetes.io/projected/52efc11e-b616-4c33-97d8-06fed7d75f82-kube-api-access-pcfz2\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.338578 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338564    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-var-lib-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.338834 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338589    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-host-slash\") pod \"iptables-alerter-f2s9d\" (UID: \"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5\") " pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:29:20.338834 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338632    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-cni-binary-copy\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.338834 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338658    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.338834 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338683    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-socket-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.338834 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338708    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-registration-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.338834 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338743    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-slash\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.338834 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338769    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.338834 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338795    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.339293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338840    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-iptables-alerter-script\") pod \"iptables-alerter-f2s9d\" (UID: \"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5\") " pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:29:20.339293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338883    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkz6g\" (UniqueName: \"kubernetes.io/projected/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-kube-api-access-gkz6g\") pod \"iptables-alerter-f2s9d\" (UID: \"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5\") " pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:29:20.339293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338908    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-dbus\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:20.339293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338932    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:20.339293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338971    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs72w\" (UniqueName: \"kubernetes.io/projected/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-kube-api-access-cs72w\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.339293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.338997    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-ovn\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.339293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339041    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-run-ovn-kubernetes\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.339293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339074    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-sys-fs\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.339293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339099    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-cni-bin\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.339293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339125    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-kubelet\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.339293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339150    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-cnibin\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.339293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339176    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da310e19-8c58-4785-a88e-09edc8a63e2e-host\") pod \"node-ca-lcxk5\" (UID: \"da310e19-8c58-4785-a88e-09edc8a63e2e\") " pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:29:20.339293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339200    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wkt8\" (UniqueName: \"kubernetes.io/projected/da310e19-8c58-4785-a88e-09edc8a63e2e-kube-api-access-4wkt8\") pod \"node-ca-lcxk5\" (UID: \"da310e19-8c58-4785-a88e-09edc8a63e2e\") " pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:29:20.339293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339228    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.339293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339255    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-device-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.339293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339288    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-env-overrides\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.340148 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339342    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-kubelet-config\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:20.340148 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339367    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-run-netns\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.340148 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339416    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-etc-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.340148 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339444    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-log-socket\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.340148 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339482    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-cni-netd\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.340148 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339511    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovnkube-config\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.340148 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339551    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-etc-selinux\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.340148 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339580    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-systemd\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.340148 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339613    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzkdg\" (UniqueName: \"kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg\") pod \"network-check-target-6wwsm\" (UID: \"926e8fa4-1a22-48b1-832e-ea924ef61178\") " pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:20.340148 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339642    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.340148 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339669    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-node-log\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.340148 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.339696    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovn-node-metrics-cert\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.341101 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.340594    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.342179 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.342158    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:20.342359 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:20.342334    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b4p" podUID="bb3df8dc-d134-4cee-b95e-21e32f1df218"
Oct 30 19:29:20.345321 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.345291    2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:29:20.368967 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.368935    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9d86s"
Oct 30 19:29:20.382673 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.382647    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9d86s"
Oct 30 19:29:20.424350 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.424281    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:29:20.424623 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.424589    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:29:20.424623 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.424597    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:29:20.424780 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.424652    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:29:20.424780 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.424735    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:29:20.424780 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.424653    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:29:20.425056 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.425039    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:29:20.425121 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.425074    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:29:20.425179 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.425127    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:29:20.425232 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.425193    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p6d49\""
Oct 30 19:29:20.425538 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.425232    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:29:20.425538 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.425288    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fbg5p\""
Oct 30 19:29:20.425538 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.425309    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z724k\""
Oct 30 19:29:20.425538 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.425326    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:29:20.425538 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.425352    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:29:20.425538 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.425399    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:29:20.425538 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.425409    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:29:20.425538 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.425421    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q2kw5\""
Oct 30 19:29:20.425932 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.425543    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pn76m\""
Oct 30 19:29:20.425932 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.425625    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2cs46\""
Oct 30 19:29:20.425932 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.425652    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:29:20.427779 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.427760    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:29:20.440426 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440392    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-kubelet\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.440608 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440438    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-run\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.440608 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440467    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-sys\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.440608 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-729fn\" (UniqueName: \"kubernetes.io/projected/e683558b-f046-46ae-99c4-4862e1247b03-kube-api-access-729fn\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.440608 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440555    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-kubelet-config\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:20.440608 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440593    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-run-netns\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.440878 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440648    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-run-netns\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.440878 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440654    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-kubelet-config\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:20.440878 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440659    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-etc-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.440878 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440698    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-etc-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.440878 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440699    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-log-socket\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.440878 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440734    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-log-socket\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.440878 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440742    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-cni-netd\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.440878 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440769    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovnkube-config\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.440878 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440800    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e683558b-f046-46ae-99c4-4862e1247b03-etc-tuned\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.440878 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440810    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-cni-netd\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.440878 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440828    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-etc-selinux\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440900    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-cni-bin\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440981    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-etc-selinux\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.440992    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-hostroot\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441035    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-conf-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441060    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-multus-certs\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441119    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441143    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-node-log\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441162    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22dvx\" (UniqueName: \"kubernetes.io/projected/c6ec634f-81f4-4d4e-97ef-33092b38052a-kube-api-access-22dvx\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441193    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5f201ec-3c4d-4738-b721-ca4e445d63ad-tmp-dir\") pod \"node-resolver-2sdcp\" (UID: \"b5f201ec-3c4d-4738-b721-ca4e445d63ad\") " pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441217    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-node-log\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441231    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-cni-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441255    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-etc-kubernetes\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441282    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-var-lib-kubelet\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441307    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da310e19-8c58-4785-a88e-09edc8a63e2e-serviceca\") pod \"node-ca-lcxk5\" (UID: \"da310e19-8c58-4785-a88e-09edc8a63e2e\") " pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441330    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcfz2\" (UniqueName: \"kubernetes.io/projected/52efc11e-b616-4c33-97d8-06fed7d75f82-kube-api-access-pcfz2\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441353    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovnkube-script-lib\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.441488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441377    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-host-slash\") pod \"iptables-alerter-f2s9d\" (UID: \"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5\") " pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:29:20.443644 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441404    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-socket-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.443644 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441427    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-registration-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.443644 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441451    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.443644 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441477    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-cni-multus\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.443644 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441503    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkz6g\" (UniqueName: \"kubernetes.io/projected/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-kube-api-access-gkz6g\") pod \"iptables-alerter-f2s9d\" (UID: \"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5\") " pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:29:20.443644 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441530    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-lib-modules\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.443644 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441554    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-host\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.443644 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441577    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-kubernetes\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.443644 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441591    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovnkube-config\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.443644 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441608    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wkt8\" (UniqueName: \"kubernetes.io/projected/da310e19-8c58-4785-a88e-09edc8a63e2e-kube-api-access-4wkt8\") pod \"node-ca-lcxk5\" (UID: \"da310e19-8c58-4785-a88e-09edc8a63e2e\") " pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:29:20.443644 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441635    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.443644 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441677    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.443644 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441775    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-host-slash\") pod \"iptables-alerter-f2s9d\" (UID: \"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5\") " pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:29:20.443644 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441774    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-registration-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.443644 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441783    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-device-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.443644 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441878    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-socket-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441950    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b5f201ec-3c4d-4738-b721-ca4e445d63ad-hosts-file\") pod \"node-resolver-2sdcp\" (UID: \"b5f201ec-3c4d-4738-b721-ca4e445d63ad\") " pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.441988    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-system-cni-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442012    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-socket-dir-parent\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442059    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72lkw\" (UniqueName: \"kubernetes.io/projected/ebaeaca4-4212-4bcc-b979-873495265bad-kube-api-access-72lkw\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442085    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-modprobe-d\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442113    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs\") pod \"network-metrics-daemon-87b4p\" (UID: \"bb3df8dc-d134-4cee-b95e-21e32f1df218\") " pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442197    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-systemd\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442235    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkdg\" (UniqueName: \"kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg\") pod \"network-check-target-6wwsm\" (UID: \"926e8fa4-1a22-48b1-832e-ea924ef61178\") " pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442263    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-netns\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442310    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da310e19-8c58-4785-a88e-09edc8a63e2e-serviceca\") pod \"node-ca-lcxk5\" (UID: \"da310e19-8c58-4785-a88e-09edc8a63e2e\") " pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442312    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysctl-conf\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442369    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovn-node-metrics-cert\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442397    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-systemd-units\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442414    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-system-cni-dir\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442443    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442452    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovnkube-script-lib\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.444442 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442462    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-var-lib-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442468    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442480    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebaeaca4-4212-4bcc-b979-873495265bad-multus-daemon-config\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442516    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-device-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442552    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-systemd\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442577    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442612    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysconfig\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442626    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-system-cni-dir\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442643    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zgxv\" (UniqueName: \"kubernetes.io/projected/bb3df8dc-d134-4cee-b95e-21e32f1df218-kube-api-access-8zgxv\") pod \"network-metrics-daemon-87b4p\" (UID: \"bb3df8dc-d134-4cee-b95e-21e32f1df218\") " pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442688    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-var-lib-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442711    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-systemd-units\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442723    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c246f08a-6fa3-4664-8a73-567d35f22d3a-agent-certs\") pod \"konnectivity-agent-ncbz9\" (UID: \"c246f08a-6fa3-4664-8a73-567d35f22d3a\") " pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442751    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-cni-binary-copy\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442766    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442778    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442854    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-slash\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442913    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.445202 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442928    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442977    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-cnibin\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442987    2569 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.442988    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443054    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-env-overrides\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443055    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-slash\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443108    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e683558b-f046-46ae-99c4-4862e1247b03-tmp\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443161    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-cni-binary-copy\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443168    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-iptables-alerter-script\") pod \"iptables-alerter-f2s9d\" (UID: \"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5\") " pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443321    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-dbus\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443404    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443465    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs72w\" (UniqueName: \"kubernetes.io/projected/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-kube-api-access-cs72w\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443557    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-env-overrides\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:20.443571    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443596    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-ovn\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:20.443719    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret podName:bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e nodeName:}" failed. No retries permitted until 2026-10-30 19:29:20.943697211 +0000 UTC m=+3.027823630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret") pod "global-pull-secret-syncer-4zdqg" (UID: "bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443716    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-iptables-alerter-script\") pod \"iptables-alerter-f2s9d\" (UID: \"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5\") " pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:29:20.445891 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443789    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-run-ovn-kubernetes\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443820    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-cni-bin\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443850    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6hxs\" (UniqueName: \"kubernetes.io/projected/b5f201ec-3c4d-4738-b721-ca4e445d63ad-kube-api-access-l6hxs\") pod \"node-resolver-2sdcp\" (UID: \"b5f201ec-3c4d-4738-b721-ca4e445d63ad\") " pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443911    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-sys-fs\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443936    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c246f08a-6fa3-4664-8a73-567d35f22d3a-konnectivity-ca\") pod \"konnectivity-agent-ncbz9\" (UID: \"c246f08a-6fa3-4664-8a73-567d35f22d3a\") " pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443962    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebaeaca4-4212-4bcc-b979-873495265bad-cni-binary-copy\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.443990    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-k8s-cni-cncf-io\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.444015    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysctl-d\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.444033    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-dbus\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.444078    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-run-ovn-kubernetes\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.444091    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-sys-fs\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.444126    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-ovn\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.444135    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-cni-bin\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.444163    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-systemd\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.444204    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-kubelet\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.444234    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-cnibin\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.444268    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da310e19-8c58-4785-a88e-09edc8a63e2e-host\") pod \"node-ca-lcxk5\" (UID: \"da310e19-8c58-4785-a88e-09edc8a63e2e\") " pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:29:20.446492 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.444350    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da310e19-8c58-4785-a88e-09edc8a63e2e-host\") pod \"node-ca-lcxk5\" (UID: \"da310e19-8c58-4785-a88e-09edc8a63e2e\") " pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:29:20.447326 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.444397    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-cnibin\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.447326 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.444404    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-kubelet\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.449714 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.449690    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovn-node-metrics-cert\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.453138 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:20.453101    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:29:20.453138 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:20.453132    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:29:20.453353 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:20.453148    2569 projected.go:196] Error preparing data for projected volume kube-api-access-rzkdg for pod openshift-network-diagnostics/network-check-target-6wwsm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:20.453353 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:20.453202    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg podName:926e8fa4-1a22-48b1-832e-ea924ef61178 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:20.953186453 +0000 UTC m=+3.037312852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rzkdg" (UniqueName: "kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg") pod "network-check-target-6wwsm" (UID: "926e8fa4-1a22-48b1-832e-ea924ef61178") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:20.456311 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.456285    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcfz2\" (UniqueName: \"kubernetes.io/projected/52efc11e-b616-4c33-97d8-06fed7d75f82-kube-api-access-pcfz2\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.456418 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.456393    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wkt8\" (UniqueName: \"kubernetes.io/projected/da310e19-8c58-4785-a88e-09edc8a63e2e-kube-api-access-4wkt8\") pod \"node-ca-lcxk5\" (UID: \"da310e19-8c58-4785-a88e-09edc8a63e2e\") " pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:29:20.457424 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.457398    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22dvx\" (UniqueName: \"kubernetes.io/projected/c6ec634f-81f4-4d4e-97ef-33092b38052a-kube-api-access-22dvx\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.457822 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.457798    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkz6g\" (UniqueName: \"kubernetes.io/projected/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-kube-api-access-gkz6g\") pod \"iptables-alerter-f2s9d\" (UID: \"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5\") " pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:29:20.466671 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.466649    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs72w\" (UniqueName: \"kubernetes.io/projected/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-kube-api-access-cs72w\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.482274 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.482203    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal" event={"ID":"f8f0f705095219e55fd87172f433b51c","Type":"ContainerStarted","Data":"5f1402c6a45b330bbb1cc8cd1e011283fcbb6dbc621e3c8679a903e3fa69bbd0"}
Oct 30 19:29:20.483577 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.483545    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal" event={"ID":"caaa86be00fc4b40c2a164ffe289a741","Type":"ContainerStarted","Data":"a7a16614b766a578de04bc5f01c9de025ad846dc68802f7bae453eef46f936e8"}
Oct 30 19:29:20.545082 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545045    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-netns\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.545266 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545089    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysctl-conf\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.545266 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545119    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebaeaca4-4212-4bcc-b979-873495265bad-multus-daemon-config\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.545266 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545143    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysconfig\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.545266 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545169    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zgxv\" (UniqueName: \"kubernetes.io/projected/bb3df8dc-d134-4cee-b95e-21e32f1df218-kube-api-access-8zgxv\") pod \"network-metrics-daemon-87b4p\" (UID: \"bb3df8dc-d134-4cee-b95e-21e32f1df218\") " pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:20.545266 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545183    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-netns\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.545266 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545193    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c246f08a-6fa3-4664-8a73-567d35f22d3a-agent-certs\") pod \"konnectivity-agent-ncbz9\" (UID: \"c246f08a-6fa3-4664-8a73-567d35f22d3a\") " pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:29:20.545266 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545241    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysconfig\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.545266 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545264    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-cnibin\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.545668 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545289    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysctl-conf\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.545668 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545293    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e683558b-f046-46ae-99c4-4862e1247b03-tmp\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.545668 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545364    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6hxs\" (UniqueName: \"kubernetes.io/projected/b5f201ec-3c4d-4738-b721-ca4e445d63ad-kube-api-access-l6hxs\") pod \"node-resolver-2sdcp\" (UID: \"b5f201ec-3c4d-4738-b721-ca4e445d63ad\") " pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:29:20.545668 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545394    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c246f08a-6fa3-4664-8a73-567d35f22d3a-konnectivity-ca\") pod \"konnectivity-agent-ncbz9\" (UID: \"c246f08a-6fa3-4664-8a73-567d35f22d3a\") " pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:29:20.545668 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545401    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-cnibin\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.545668 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545421    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebaeaca4-4212-4bcc-b979-873495265bad-cni-binary-copy\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.545668 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545448    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-k8s-cni-cncf-io\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.545668 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545469    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysctl-d\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.545668 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545491    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-systemd\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.545668 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545519    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-kubelet\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.545668 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545544    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-run\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.545668 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545548    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-k8s-cni-cncf-io\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.545668 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545567    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-sys\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.545668 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545654    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-729fn\" (UniqueName: \"kubernetes.io/projected/e683558b-f046-46ae-99c4-4862e1247b03-kube-api-access-729fn\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545689    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e683558b-f046-46ae-99c4-4862e1247b03-etc-tuned\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545714    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-cni-bin\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545738    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-hostroot\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545764    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-conf-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545784    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-sys\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545849    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-run\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545878    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysctl-d\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545970    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-systemd\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545789    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-multus-certs\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545885    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebaeaca4-4212-4bcc-b979-873495265bad-multus-daemon-config\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546022    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5f201ec-3c4d-4738-b721-ca4e445d63ad-tmp-dir\") pod \"node-resolver-2sdcp\" (UID: \"b5f201ec-3c4d-4738-b721-ca4e445d63ad\") " pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546049    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-multus-certs\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546065    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-cni-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546093    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-etc-kubernetes\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546100    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-hostroot\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.545734    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-kubelet\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546139    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-var-lib-kubelet\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.546402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546138    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-cni-bin\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546170    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c246f08a-6fa3-4664-8a73-567d35f22d3a-konnectivity-ca\") pod \"konnectivity-agent-ncbz9\" (UID: \"c246f08a-6fa3-4664-8a73-567d35f22d3a\") " pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546174    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-cni-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546177    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-cni-multus\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546206    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebaeaca4-4212-4bcc-b979-873495265bad-cni-binary-copy\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546226    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-cni-multus\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546202    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-etc-kubernetes\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546239    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-conf-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546256    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-lib-modules\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546244    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-var-lib-kubelet\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546297    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-host\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546326    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-kubernetes\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546358    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-host\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546370    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-lib-modules\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546361    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b5f201ec-3c4d-4738-b721-ca4e445d63ad-hosts-file\") pod \"node-resolver-2sdcp\" (UID: \"b5f201ec-3c4d-4738-b721-ca4e445d63ad\") " pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546411    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-kubernetes\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546412    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-system-cni-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546416    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b5f201ec-3c4d-4738-b721-ca4e445d63ad-hosts-file\") pod \"node-resolver-2sdcp\" (UID: \"b5f201ec-3c4d-4738-b721-ca4e445d63ad\") " pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:29:20.547316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546444    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-socket-dir-parent\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.547896 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546460    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-system-cni-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.547896 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546473    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72lkw\" (UniqueName: \"kubernetes.io/projected/ebaeaca4-4212-4bcc-b979-873495265bad-kube-api-access-72lkw\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.547896 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546503    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-modprobe-d\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.547896 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546521    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5f201ec-3c4d-4738-b721-ca4e445d63ad-tmp-dir\") pod \"node-resolver-2sdcp\" (UID: \"b5f201ec-3c4d-4738-b721-ca4e445d63ad\") " pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:29:20.547896 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546531    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs\") pod \"network-metrics-daemon-87b4p\" (UID: \"bb3df8dc-d134-4cee-b95e-21e32f1df218\") " pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:20.547896 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546527    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-socket-dir-parent\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.547896 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.546652    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-modprobe-d\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.547896 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:20.546676    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:20.547896 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:20.546766    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs podName:bb3df8dc-d134-4cee-b95e-21e32f1df218 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:21.046739513 +0000 UTC m=+3.130865912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs") pod "network-metrics-daemon-87b4p" (UID: "bb3df8dc-d134-4cee-b95e-21e32f1df218") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:20.548709 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.548686    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c246f08a-6fa3-4664-8a73-567d35f22d3a-agent-certs\") pod \"konnectivity-agent-ncbz9\" (UID: \"c246f08a-6fa3-4664-8a73-567d35f22d3a\") " pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:29:20.551013 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.550271    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e683558b-f046-46ae-99c4-4862e1247b03-etc-tuned\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.551013 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.550291    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e683558b-f046-46ae-99c4-4862e1247b03-tmp\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.561759 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.561680    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6hxs\" (UniqueName: \"kubernetes.io/projected/b5f201ec-3c4d-4738-b721-ca4e445d63ad-kube-api-access-l6hxs\") pod \"node-resolver-2sdcp\" (UID: \"b5f201ec-3c4d-4738-b721-ca4e445d63ad\") " pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:29:20.563091 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.563064    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zgxv\" (UniqueName: \"kubernetes.io/projected/bb3df8dc-d134-4cee-b95e-21e32f1df218-kube-api-access-8zgxv\") pod \"network-metrics-daemon-87b4p\" (UID: \"bb3df8dc-d134-4cee-b95e-21e32f1df218\") " pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:20.565211 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.565184    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-729fn\" (UniqueName: \"kubernetes.io/projected/e683558b-f046-46ae-99c4-4862e1247b03-kube-api-access-729fn\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.566459 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.566434    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72lkw\" (UniqueName: \"kubernetes.io/projected/ebaeaca4-4212-4bcc-b979-873495265bad-kube-api-access-72lkw\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.630652 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.630601    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:29:20.644967 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.641367    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:29:20.644967 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:20.641485    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda310e19_8c58_4785_a88e_09edc8a63e2e.slice/crio-17bebd71442cee57df0a63815d5d2ae487a270b957f1bfe226ce7d9475b88fe3 WatchSource:0}: Error finding container 17bebd71442cee57df0a63815d5d2ae487a270b957f1bfe226ce7d9475b88fe3: Status 404 returned error can't find the container with id 17bebd71442cee57df0a63815d5d2ae487a270b957f1bfe226ce7d9475b88fe3
Oct 30 19:29:20.653036 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:20.653004    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e7bd96_2bff_4a9c_9c6d_2a1e555975f5.slice/crio-7cdfd04130d7c43badbb4acabd61789fcc62d63f0b04bade27b9fcd4876c1cdb WatchSource:0}: Error finding container 7cdfd04130d7c43badbb4acabd61789fcc62d63f0b04bade27b9fcd4876c1cdb: Status 404 returned error can't find the container with id 7cdfd04130d7c43badbb4acabd61789fcc62d63f0b04bade27b9fcd4876c1cdb
Oct 30 19:29:20.673564 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.673256    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:29:20.684296 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.684264    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:29:20.686052 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:20.686014    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52efc11e_b616_4c33_97d8_06fed7d75f82.slice/crio-0166d3b7fd00652d2525020761ad62349e7fcd17cb6c5866bd5633d9827af5f5 WatchSource:0}: Error finding container 0166d3b7fd00652d2525020761ad62349e7fcd17cb6c5866bd5633d9827af5f5: Status 404 returned error can't find the container with id 0166d3b7fd00652d2525020761ad62349e7fcd17cb6c5866bd5633d9827af5f5
Oct 30 19:29:20.687106 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.687082    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:29:20.687730 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.687708    2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:29:20.694777 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.694750    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:29:20.699569 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.699395    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:20.700855 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:20.700828    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode683558b_f046_46ae_99c4_4862e1247b03.slice/crio-a159f89e1eb5008d183ff76861d00f8088915da54ceea0cc0074c4be0270810a WatchSource:0}: Error finding container a159f89e1eb5008d183ff76861d00f8088915da54ceea0cc0074c4be0270810a: Status 404 returned error can't find the container with id a159f89e1eb5008d183ff76861d00f8088915da54ceea0cc0074c4be0270810a
Oct 30 19:29:20.703762 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.703200    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:29:20.706012 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.705989    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k6rd4"
Oct 30 19:29:20.718610 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:20.718576    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f201ec_3c4d_4738_b721_ca4e445d63ad.slice/crio-89878ff87d7d6936af6e057d953f2858ab03f66e9789b28558a113c376fe20d5 WatchSource:0}: Error finding container 89878ff87d7d6936af6e057d953f2858ab03f66e9789b28558a113c376fe20d5: Status 404 returned error can't find the container with id 89878ff87d7d6936af6e057d953f2858ab03f66e9789b28558a113c376fe20d5
Oct 30 19:29:20.719665 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:20.719626    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ec634f_81f4_4d4e_97ef_33092b38052a.slice/crio-f87225c4e0d6745d6509a153c22157103d8b00e9661b92b940d95e3fa9abbf23 WatchSource:0}: Error finding container f87225c4e0d6745d6509a153c22157103d8b00e9661b92b940d95e3fa9abbf23: Status 404 returned error can't find the container with id f87225c4e0d6745d6509a153c22157103d8b00e9661b92b940d95e3fa9abbf23
Oct 30 19:29:20.725953 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:20.725889    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc246f08a_6fa3_4664_8a73_567d35f22d3a.slice/crio-343d60461be4bd745b9eb08058f432ffb36a1cc7bf0f4822efb0101a0359b6de WatchSource:0}: Error finding container 343d60461be4bd745b9eb08058f432ffb36a1cc7bf0f4822efb0101a0359b6de: Status 404 returned error can't find the container with id 343d60461be4bd745b9eb08058f432ffb36a1cc7bf0f4822efb0101a0359b6de
Oct 30 19:29:20.726561 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:20.726541    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebaeaca4_4212_4bcc_b979_873495265bad.slice/crio-14ac690ea497ca589a3c1266822bb4e2e626396b5b84648b4d25e101985876f9 WatchSource:0}: Error finding container 14ac690ea497ca589a3c1266822bb4e2e626396b5b84648b4d25e101985876f9: Status 404 returned error can't find the container with id 14ac690ea497ca589a3c1266822bb4e2e626396b5b84648b4d25e101985876f9
Oct 30 19:29:20.950243 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:20.950140    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:20.950400 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:20.950330    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:20.950445 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:20.950420    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret podName:bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e nodeName:}" failed. No retries permitted until 2026-10-30 19:29:21.950396313 +0000 UTC m=+4.034522724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret") pod "global-pull-secret-syncer-4zdqg" (UID: "bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:21.050529 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:21.050484    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs\") pod \"network-metrics-daemon-87b4p\" (UID: \"bb3df8dc-d134-4cee-b95e-21e32f1df218\") " pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:21.050529 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:21.050542    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkdg\" (UniqueName: \"kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg\") pod \"network-check-target-6wwsm\" (UID: \"926e8fa4-1a22-48b1-832e-ea924ef61178\") " pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:21.050789 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:21.050681    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:21.050789 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:21.050729    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:29:21.050789 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:21.050748    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:29:21.050789 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:21.050756    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs podName:bb3df8dc-d134-4cee-b95e-21e32f1df218 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:22.050733147 +0000 UTC m=+4.134859557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs") pod "network-metrics-daemon-87b4p" (UID: "bb3df8dc-d134-4cee-b95e-21e32f1df218") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:21.050789 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:21.050762    2569 projected.go:196] Error preparing data for projected volume kube-api-access-rzkdg for pod openshift-network-diagnostics/network-check-target-6wwsm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:21.051106 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:21.050814    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg podName:926e8fa4-1a22-48b1-832e-ea924ef61178 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:22.050796982 +0000 UTC m=+4.134923386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rzkdg" (UniqueName: "kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg") pod "network-check-target-6wwsm" (UID: "926e8fa4-1a22-48b1-832e-ea924ef61178") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:21.387180 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:21.387030    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:24:20 +0000 UTC" deadline="2028-04-27 05:16:44.448830921 +0000 UTC"
Oct 30 19:29:21.387180 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:21.387102    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13065h47m23.061733153s"
Oct 30 19:29:21.498762 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:21.498657    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7" event={"ID":"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c","Type":"ContainerStarted","Data":"339613c0d5b9ef3b19a2724086dfcd73a391e38e746e7a49802e28c44478c19d"}
Oct 30 19:29:21.519285 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:21.519194    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg8f2" event={"ID":"52efc11e-b616-4c33-97d8-06fed7d75f82","Type":"ContainerStarted","Data":"0166d3b7fd00652d2525020761ad62349e7fcd17cb6c5866bd5633d9827af5f5"}
Oct 30 19:29:21.530150 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:21.530070    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lcxk5" event={"ID":"da310e19-8c58-4785-a88e-09edc8a63e2e","Type":"ContainerStarted","Data":"17bebd71442cee57df0a63815d5d2ae487a270b957f1bfe226ce7d9475b88fe3"}
Oct 30 19:29:21.549637 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:21.549585    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k6rd4" event={"ID":"ebaeaca4-4212-4bcc-b979-873495265bad","Type":"ContainerStarted","Data":"14ac690ea497ca589a3c1266822bb4e2e626396b5b84648b4d25e101985876f9"}
Oct 30 19:29:21.569472 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:21.569366    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2sdcp" event={"ID":"b5f201ec-3c4d-4738-b721-ca4e445d63ad","Type":"ContainerStarted","Data":"89878ff87d7d6936af6e057d953f2858ab03f66e9789b28558a113c376fe20d5"}
Oct 30 19:29:21.597690 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:21.597537    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fcz7q" event={"ID":"e683558b-f046-46ae-99c4-4862e1247b03","Type":"ContainerStarted","Data":"a159f89e1eb5008d183ff76861d00f8088915da54ceea0cc0074c4be0270810a"}
Oct 30 19:29:21.621773 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:21.621727    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f2s9d" event={"ID":"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5","Type":"ContainerStarted","Data":"7cdfd04130d7c43badbb4acabd61789fcc62d63f0b04bade27b9fcd4876c1cdb"}
Oct 30 19:29:21.625108 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:21.625072    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ncbz9" event={"ID":"c246f08a-6fa3-4664-8a73-567d35f22d3a","Type":"ContainerStarted","Data":"343d60461be4bd745b9eb08058f432ffb36a1cc7bf0f4822efb0101a0359b6de"}
Oct 30 19:29:21.634361 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:21.634319    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h" event={"ID":"c6ec634f-81f4-4d4e-97ef-33092b38052a","Type":"ContainerStarted","Data":"f87225c4e0d6745d6509a153c22157103d8b00e9661b92b940d95e3fa9abbf23"}
Oct 30 19:29:21.960137 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:21.960066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:21.960366 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:21.960347    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:21.960479 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:21.960459    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret podName:bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e nodeName:}" failed. No retries permitted until 2026-10-30 19:29:23.960439714 +0000 UTC m=+6.044566115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret") pod "global-pull-secret-syncer-4zdqg" (UID: "bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:22.061409 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:22.061360    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs\") pod \"network-metrics-daemon-87b4p\" (UID: \"bb3df8dc-d134-4cee-b95e-21e32f1df218\") " pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:22.061601 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:22.061420    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkdg\" (UniqueName: \"kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg\") pod \"network-check-target-6wwsm\" (UID: \"926e8fa4-1a22-48b1-832e-ea924ef61178\") " pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:22.061601 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:22.061572    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:29:22.061601 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:22.061592    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:29:22.061755 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:22.061605    2569 projected.go:196] Error preparing data for projected volume kube-api-access-rzkdg for pod openshift-network-diagnostics/network-check-target-6wwsm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:22.061755 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:22.061665    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg podName:926e8fa4-1a22-48b1-832e-ea924ef61178 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:24.061646997 +0000 UTC m=+6.145773407 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rzkdg" (UniqueName: "kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg") pod "network-check-target-6wwsm" (UID: "926e8fa4-1a22-48b1-832e-ea924ef61178") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:22.062137 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:22.062113    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:22.062234 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:22.062179    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs podName:bb3df8dc-d134-4cee-b95e-21e32f1df218 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:24.06216154 +0000 UTC m=+6.146287951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs") pod "network-metrics-daemon-87b4p" (UID: "bb3df8dc-d134-4cee-b95e-21e32f1df218") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:22.387673 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:22.387268    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:24:20 +0000 UTC" deadline="2028-03-30 06:43:34.587572338 +0000 UTC"
Oct 30 19:29:22.387673 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:22.387597    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12395h14m12.199993152s"
Oct 30 19:29:22.478551 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:22.477919    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:22.478551 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:22.478046    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b4p" podUID="bb3df8dc-d134-4cee-b95e-21e32f1df218"
Oct 30 19:29:22.478551 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:22.478371    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:22.478551 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:22.478432    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zdqg" podUID="bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e"
Oct 30 19:29:22.478551 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:22.478471    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:22.478551 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:22.478513    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wwsm" podUID="926e8fa4-1a22-48b1-832e-ea924ef61178"
Oct 30 19:29:22.642434 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:22.642302    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal" event={"ID":"f8f0f705095219e55fd87172f433b51c","Type":"ContainerStarted","Data":"6dc3984086e19e0242c772c73e7850ffe588ce8ddf18e52e7ea205a9fa942ee1"}
Oct 30 19:29:22.648286 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:22.648250    2569 generic.go:356] "Generic (PLEG): container finished" podID="caaa86be00fc4b40c2a164ffe289a741" containerID="892abd471688048e00426fbf286e327f6b79de36e04178ae2fa217ef29dd98e6" exitCode=0
Oct 30 19:29:22.648452 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:22.648309    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal" event={"ID":"caaa86be00fc4b40c2a164ffe289a741","Type":"ContainerDied","Data":"892abd471688048e00426fbf286e327f6b79de36e04178ae2fa217ef29dd98e6"}
Oct 30 19:29:22.661204 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:22.660535    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal" podStartSLOduration=2.660516331 podStartE2EDuration="2.660516331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:22.65994884 +0000 UTC m=+4.744075262" watchObservedRunningTime="2026-10-30 19:29:22.660516331 +0000 UTC m=+4.744642753"
Oct 30 19:29:23.681274 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:23.681195    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal" event={"ID":"caaa86be00fc4b40c2a164ffe289a741","Type":"ContainerStarted","Data":"edd8040984c746349d0e39f87860025a1d3288332af63fdf436bdcf7d09f100b"}
Oct 30 19:29:23.701679 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:23.701614    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal" podStartSLOduration=3.701592559 podStartE2EDuration="3.701592559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:23.701532873 +0000 UTC m=+5.785659292" watchObservedRunningTime="2026-10-30 19:29:23.701592559 +0000 UTC m=+5.785718981"
Oct 30 19:29:23.976794 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:23.976751    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:23.976999 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:23.976944    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:23.977063 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:23.977015    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret podName:bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e nodeName:}" failed. No retries permitted until 2026-10-30 19:29:27.97699564 +0000 UTC m=+10.061122054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret") pod "global-pull-secret-syncer-4zdqg" (UID: "bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:24.078030 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:24.077297    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs\") pod \"network-metrics-daemon-87b4p\" (UID: \"bb3df8dc-d134-4cee-b95e-21e32f1df218\") " pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:24.078030 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:24.077355    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkdg\" (UniqueName: \"kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg\") pod \"network-check-target-6wwsm\" (UID: \"926e8fa4-1a22-48b1-832e-ea924ef61178\") " pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:24.078030 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:24.077543    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:29:24.078030 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:24.077562    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:29:24.078030 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:24.077576    2569 projected.go:196] Error preparing data for projected volume kube-api-access-rzkdg for pod openshift-network-diagnostics/network-check-target-6wwsm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:24.078030 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:24.077648    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg podName:926e8fa4-1a22-48b1-832e-ea924ef61178 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:28.077628241 +0000 UTC m=+10.161754642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rzkdg" (UniqueName: "kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg") pod "network-check-target-6wwsm" (UID: "926e8fa4-1a22-48b1-832e-ea924ef61178") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:24.078503 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:24.078085    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:24.078503 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:24.078149    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs podName:bb3df8dc-d134-4cee-b95e-21e32f1df218 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:28.078131765 +0000 UTC m=+10.162258179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs") pod "network-metrics-daemon-87b4p" (UID: "bb3df8dc-d134-4cee-b95e-21e32f1df218") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:24.478138 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:24.478101    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:24.478324 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:24.478095    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:24.478324 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:24.478249    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:24.478471 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:24.478323    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zdqg" podUID="bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e"
Oct 30 19:29:24.478471 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:24.478394    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b4p" podUID="bb3df8dc-d134-4cee-b95e-21e32f1df218"
Oct 30 19:29:24.478471 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:24.478245    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wwsm" podUID="926e8fa4-1a22-48b1-832e-ea924ef61178"
Oct 30 19:29:24.688754 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:24.688712    2569 generic.go:356] "Generic (PLEG): container finished" podID="52efc11e-b616-4c33-97d8-06fed7d75f82" containerID="c09c9f078176b179f93efd4d733d052b91d36929bb828c32c63507031f6fa9ed" exitCode=0
Oct 30 19:29:24.689450 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:24.689423    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg8f2" event={"ID":"52efc11e-b616-4c33-97d8-06fed7d75f82","Type":"ContainerDied","Data":"c09c9f078176b179f93efd4d733d052b91d36929bb828c32c63507031f6fa9ed"}
Oct 30 19:29:25.699016 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:25.698928    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ncbz9" event={"ID":"c246f08a-6fa3-4664-8a73-567d35f22d3a","Type":"ContainerStarted","Data":"c8a7e5c9c868abe17990f06eef3bbb10450bb11b1c41a664f438cf8d6db81072"}
Oct 30 19:29:26.478730 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:26.477975    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:26.478730 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:26.478123    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b4p" podUID="bb3df8dc-d134-4cee-b95e-21e32f1df218"
Oct 30 19:29:26.478730 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:26.478486    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:26.478730 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:26.478583    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zdqg" podUID="bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e"
Oct 30 19:29:26.478730 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:26.478624    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:26.478730 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:26.478688    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wwsm" podUID="926e8fa4-1a22-48b1-832e-ea924ef61178"
Oct 30 19:29:26.709515 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:26.709394    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lcxk5" event={"ID":"da310e19-8c58-4785-a88e-09edc8a63e2e","Type":"ContainerStarted","Data":"d4a266d6da44f723fb87848dceb8649f5564298646aa708b4b5cbe80a99a3ea4"}
Oct 30 19:29:26.726349 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:26.726290    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ncbz9" podStartSLOduration=3.170187599 podStartE2EDuration="7.726271411s" totalImagesPullingTime="4.556083812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:25.718333861 +0000 UTC m=+7.802460278" watchObservedRunningTime="2026-10-30 19:29:26.726271411 +0000 UTC m=+8.810397827"
Oct 30 19:29:27.717130 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:27.717046    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7" event={"ID":"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c","Type":"ContainerStarted","Data":"5cba37b8b9c9eb4f5c5436a58c1f76a0030d8351fb3fcb109a6037f8cc3fb89f"}
Oct 30 19:29:28.013798 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:28.013117    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:28.013798 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:28.013334    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:28.013798 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:28.013396    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret podName:bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e nodeName:}" failed. No retries permitted until 2026-10-30 19:29:36.013378914 +0000 UTC m=+18.097505309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret") pod "global-pull-secret-syncer-4zdqg" (UID: "bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:28.115312 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:28.114465    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs\") pod \"network-metrics-daemon-87b4p\" (UID: \"bb3df8dc-d134-4cee-b95e-21e32f1df218\") " pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:28.115312 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:28.114523    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkdg\" (UniqueName: \"kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg\") pod \"network-check-target-6wwsm\" (UID: \"926e8fa4-1a22-48b1-832e-ea924ef61178\") " pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:28.115312 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:28.114717    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:29:28.115312 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:28.114741    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:29:28.115312 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:28.114755    2569 projected.go:196] Error preparing data for projected volume kube-api-access-rzkdg for pod openshift-network-diagnostics/network-check-target-6wwsm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:28.115312 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:28.114815    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg podName:926e8fa4-1a22-48b1-832e-ea924ef61178 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:36.114798717 +0000 UTC m=+18.198925115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rzkdg" (UniqueName: "kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg") pod "network-check-target-6wwsm" (UID: "926e8fa4-1a22-48b1-832e-ea924ef61178") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:28.115312 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:28.115224    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:28.115312 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:28.115274    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs podName:bb3df8dc-d134-4cee-b95e-21e32f1df218 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:36.115257868 +0000 UTC m=+18.199384275 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs") pod "network-metrics-daemon-87b4p" (UID: "bb3df8dc-d134-4cee-b95e-21e32f1df218") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:28.479501 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:28.479466    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:28.479645 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:28.479595    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wwsm" podUID="926e8fa4-1a22-48b1-832e-ea924ef61178"
Oct 30 19:29:28.480054 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:28.480026    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:28.480165 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:28.480140    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b4p" podUID="bb3df8dc-d134-4cee-b95e-21e32f1df218"
Oct 30 19:29:28.480229 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:28.480205    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:28.480296 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:28.480278    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zdqg" podUID="bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e"
Oct 30 19:29:28.609710 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:28.609664    2569 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:29:28.650939 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:28.650901    2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:29:28.652238 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:28.652021    2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:29:28.672751 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:28.672689    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lcxk5" podStartSLOduration=4.488683759 podStartE2EDuration="9.672666455s" totalImagesPullingTime="5.183982696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:26.727504957 +0000 UTC m=+8.811631374" watchObservedRunningTime="2026-10-30 19:29:28.672666455 +0000 UTC m=+10.756792873"
Oct 30 19:29:28.727792 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:28.727756    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7" event={"ID":"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c","Type":"ContainerStarted","Data":"6f47a6e2b6fcab4b9ac41ec8b8469881c81ec46c0c3cd2e38b4b0a65a0ede98a"}
Oct 30 19:29:29.403167 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:29.402370    2569 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:29:28.609880727Z","UUID":"83619f32-adbe-40cb-92ac-f02cc17bd9ce","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:29:29.405340 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:29.404778    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:29:29.405340 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:29.404829    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:29:29.731465 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:29.731374    2569 generic.go:356] "Generic (PLEG): container finished" podID="52efc11e-b616-4c33-97d8-06fed7d75f82" containerID="12aef3ec85422a735bd9e555ad20b02cd2ae7a82869b0afe7cf7c4b7eeea51a5" exitCode=0
Oct 30 19:29:29.731833 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:29.731462    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg8f2" event={"ID":"52efc11e-b616-4c33-97d8-06fed7d75f82","Type":"ContainerDied","Data":"12aef3ec85422a735bd9e555ad20b02cd2ae7a82869b0afe7cf7c4b7eeea51a5"}
Oct 30 19:29:29.733927 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:29.733136    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k6rd4" event={"ID":"ebaeaca4-4212-4bcc-b979-873495265bad","Type":"ContainerStarted","Data":"4ad92c5e5230f46ad474aa4f6630d61f0cd6682404aa59e2f1dccb47d65bbca1"}
Oct 30 19:29:29.735484 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:29.735457    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2sdcp" event={"ID":"b5f201ec-3c4d-4738-b721-ca4e445d63ad","Type":"ContainerStarted","Data":"527a839ba2ab4963d7d045c7fd166403b53fea59469f263683a024d10c556180"}
Oct 30 19:29:29.772097 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:29.772031    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2sdcp" podStartSLOduration=2.182648267 podStartE2EDuration="10.772018431s" totalImagesPullingTime="8.589370164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:29.771426015 +0000 UTC m=+11.855552431" watchObservedRunningTime="2026-10-30 19:29:29.772018431 +0000 UTC m=+11.856144846"
Oct 30 19:29:29.790687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:29.790634    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-k6rd4" podStartSLOduration=2.825289354 podStartE2EDuration="10.790619876s" totalImagesPullingTime="7.965330522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:29.790211808 +0000 UTC m=+11.874338225" watchObservedRunningTime="2026-10-30 19:29:29.790619876 +0000 UTC m=+11.874746291"
Oct 30 19:29:30.480314 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:30.476612    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:30.480314 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:30.479277    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wwsm" podUID="926e8fa4-1a22-48b1-832e-ea924ef61178"
Oct 30 19:29:30.480314 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:30.479659    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:30.480314 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:30.479765    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zdqg" podUID="bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e"
Oct 30 19:29:30.480314 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:30.480157    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:30.480314 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:30.480247    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b4p" podUID="bb3df8dc-d134-4cee-b95e-21e32f1df218"
Oct 30 19:29:30.739622 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:30.739539    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f2s9d" event={"ID":"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5","Type":"ContainerStarted","Data":"a280b4e74856ed4e60ba124f983d7ac19dd06c636360a1756774fa94ea13d105"}
Oct 30 19:29:30.742153 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:30.742122    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7" event={"ID":"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c","Type":"ContainerStarted","Data":"32e98398b619af57e1e56b097c7f125c5a2ec7b0a3f815cfa22e969c5c272619"}
Oct 30 19:29:30.745376 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:30.745346    2569 generic.go:356] "Generic (PLEG): container finished" podID="52efc11e-b616-4c33-97d8-06fed7d75f82" containerID="dc1cea409805840974e4cbbf48f28fd2632ab8388baeb4ec2238ce3ca7fe579e" exitCode=0
Oct 30 19:29:30.745468 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:30.745424    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg8f2" event={"ID":"52efc11e-b616-4c33-97d8-06fed7d75f82","Type":"ContainerDied","Data":"dc1cea409805840974e4cbbf48f28fd2632ab8388baeb4ec2238ce3ca7fe579e"}
Oct 30 19:29:30.757562 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:30.757483    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f2s9d" podStartSLOduration=3.272935946 podStartE2EDuration="11.757464937s" totalImagesPullingTime="8.484528991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:30.755671342 +0000 UTC m=+12.839797755" watchObservedRunningTime="2026-10-30 19:29:30.757464937 +0000 UTC m=+12.841591362"
Oct 30 19:29:30.799073 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:30.799016    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7" podStartSLOduration=3.175074607 podStartE2EDuration="11.799000872s" totalImagesPullingTime="8.623926265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:30.798306409 +0000 UTC m=+12.882432849" watchObservedRunningTime="2026-10-30 19:29:30.799000872 +0000 UTC m=+12.883127288"
Oct 30 19:29:31.748663 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:31.748625    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fcz7q" event={"ID":"e683558b-f046-46ae-99c4-4862e1247b03","Type":"ContainerStarted","Data":"4e30ef7c11436f79f5fc6346c23e5dac1ab94bbbf5678908af6a962b01cc9a68"}
Oct 30 19:29:31.768597 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:31.768545    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fcz7q" podStartSLOduration=2.173944742 podStartE2EDuration="12.768529599s" totalImagesPullingTime="10.594584857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:31.768012945 +0000 UTC m=+13.852139360" watchObservedRunningTime="2026-10-30 19:29:31.768529599 +0000 UTC m=+13.852656014"
Oct 30 19:29:32.476625 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:32.476587    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:32.476825 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:32.476587    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:32.476825 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:32.476704    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wwsm" podUID="926e8fa4-1a22-48b1-832e-ea924ef61178"
Oct 30 19:29:32.476954 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:32.476815    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b4p" podUID="bb3df8dc-d134-4cee-b95e-21e32f1df218"
Oct 30 19:29:32.476954 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:32.476608    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:32.477052 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:32.476951    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zdqg" podUID="bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e"
Oct 30 19:29:32.753371 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:32.753288    2569 generic.go:356] "Generic (PLEG): container finished" podID="52efc11e-b616-4c33-97d8-06fed7d75f82" containerID="1968fe429b56d461a8f4b20d4f06923d4a6a168d63b2c84a49ed69c4572ee2cd" exitCode=0
Oct 30 19:29:32.753967 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:32.753377    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg8f2" event={"ID":"52efc11e-b616-4c33-97d8-06fed7d75f82","Type":"ContainerDied","Data":"1968fe429b56d461a8f4b20d4f06923d4a6a168d63b2c84a49ed69c4572ee2cd"}
Oct 30 19:29:34.476518 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:34.476477    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:34.477116 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:34.476495    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:34.477116 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:34.476621    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:34.477116 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:34.476617    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wwsm" podUID="926e8fa4-1a22-48b1-832e-ea924ef61178"
Oct 30 19:29:34.477116 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:34.476735    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b4p" podUID="bb3df8dc-d134-4cee-b95e-21e32f1df218"
Oct 30 19:29:34.477116 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:34.476830    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zdqg" podUID="bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e"
Oct 30 19:29:34.761282 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:34.761067    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg8f2" event={"ID":"52efc11e-b616-4c33-97d8-06fed7d75f82","Type":"ContainerStarted","Data":"c3963735a2d43d376259775b214a31cedebd80ccb6b7bc6bd0b62aaabc65fc62"}
Oct 30 19:29:35.766180 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:35.766141    2569 generic.go:356] "Generic (PLEG): container finished" podID="52efc11e-b616-4c33-97d8-06fed7d75f82" containerID="c3963735a2d43d376259775b214a31cedebd80ccb6b7bc6bd0b62aaabc65fc62" exitCode=0
Oct 30 19:29:35.766656 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:35.766202    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg8f2" event={"ID":"52efc11e-b616-4c33-97d8-06fed7d75f82","Type":"ContainerDied","Data":"c3963735a2d43d376259775b214a31cedebd80ccb6b7bc6bd0b62aaabc65fc62"}
Oct 30 19:29:36.073167 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:36.072967    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:36.073167 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:36.073117    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:36.073326 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:36.073225    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret podName:bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e nodeName:}" failed. No retries permitted until 2026-10-30 19:29:52.073211178 +0000 UTC m=+34.157337571 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret") pod "global-pull-secret-syncer-4zdqg" (UID: "bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:29:36.173553 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:36.173522    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs\") pod \"network-metrics-daemon-87b4p\" (UID: \"bb3df8dc-d134-4cee-b95e-21e32f1df218\") " pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:36.173553 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:36.173557    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkdg\" (UniqueName: \"kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg\") pod \"network-check-target-6wwsm\" (UID: \"926e8fa4-1a22-48b1-832e-ea924ef61178\") " pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:36.173743 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:36.173673    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:29:36.173743 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:36.173677    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:36.173743 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:36.173740    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs podName:bb3df8dc-d134-4cee-b95e-21e32f1df218 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:52.173726102 +0000 UTC m=+34.257852509 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs") pod "network-metrics-daemon-87b4p" (UID: "bb3df8dc-d134-4cee-b95e-21e32f1df218") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:29:36.173837 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:36.173689    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:29:36.173837 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:36.173785    2569 projected.go:196] Error preparing data for projected volume kube-api-access-rzkdg for pod openshift-network-diagnostics/network-check-target-6wwsm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:36.173934 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:36.173837    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg podName:926e8fa4-1a22-48b1-832e-ea924ef61178 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:52.173823446 +0000 UTC m=+34.257949843 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rzkdg" (UniqueName: "kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg") pod "network-check-target-6wwsm" (UID: "926e8fa4-1a22-48b1-832e-ea924ef61178") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:29:36.477046 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:36.476955    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:36.477046 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:36.476973    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:36.477242 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:36.477068    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zdqg" podUID="bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e"
Oct 30 19:29:36.477242 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:36.477163    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b4p" podUID="bb3df8dc-d134-4cee-b95e-21e32f1df218"
Oct 30 19:29:36.477242 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:36.477202    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:36.477338 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:36.477269    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wwsm" podUID="926e8fa4-1a22-48b1-832e-ea924ef61178"
Oct 30 19:29:36.771598 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:36.771513    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h" event={"ID":"c6ec634f-81f4-4d4e-97ef-33092b38052a","Type":"ContainerStarted","Data":"dd9f49cd6a4b9225d9f96898448a9191290c8eeb1c48953ca9797e9bacc10c82"}
Oct 30 19:29:36.772002 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:36.771604    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h" event={"ID":"c6ec634f-81f4-4d4e-97ef-33092b38052a","Type":"ContainerStarted","Data":"eecc29d4bccfd833f50ba6be7d8bd3ae2266b83b25baee68f2f4d8e44804b02d"}
Oct 30 19:29:36.772002 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:36.771658    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h" event={"ID":"c6ec634f-81f4-4d4e-97ef-33092b38052a","Type":"ContainerStarted","Data":"ad7b639749be833ab9dea830f93ae69208ed91d800e5ab708bbbe58b79a54b8f"}
Oct 30 19:29:36.772002 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:36.771704    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h" event={"ID":"c6ec634f-81f4-4d4e-97ef-33092b38052a","Type":"ContainerStarted","Data":"cc8586c0a95f1fe5b5134cac5d1da0c35e1622d38dc1a32cf73dba01c96c79a0"}
Oct 30 19:29:36.772002 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:36.771752    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h" event={"ID":"c6ec634f-81f4-4d4e-97ef-33092b38052a","Type":"ContainerStarted","Data":"28c64efcc46d3b73cde83fdc91ccc9b2499ecec14bf752e897779acaf8dfce0c"}
Oct 30 19:29:36.772002 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:36.771786    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h" event={"ID":"c6ec634f-81f4-4d4e-97ef-33092b38052a","Type":"ContainerStarted","Data":"556fa7aadebb4dddf26b119e5df57dd9bd0572c47acde98ba5666c3ad8a0fda1"}
Oct 30 19:29:36.774211 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:36.774186    2569 generic.go:356] "Generic (PLEG): container finished" podID="52efc11e-b616-4c33-97d8-06fed7d75f82" containerID="ca51af1c99614bf904f82b0edc64206e87412719caf2cbd878749057c78a3357" exitCode=0
Oct 30 19:29:36.774337 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:36.774232    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg8f2" event={"ID":"52efc11e-b616-4c33-97d8-06fed7d75f82","Type":"ContainerDied","Data":"ca51af1c99614bf904f82b0edc64206e87412719caf2cbd878749057c78a3357"}
Oct 30 19:29:37.779502 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:37.779466    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wg8f2" event={"ID":"52efc11e-b616-4c33-97d8-06fed7d75f82","Type":"ContainerStarted","Data":"1705f50fcca17973bdfa07ac09bd3f108b44d3a1d8b49268059d602c246d4a5f"}
Oct 30 19:29:37.805884 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:37.805796    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wg8f2" podStartSLOduration=8.059578515 podStartE2EDuration="18.805778644s" totalImagesPullingTime="10.746200129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:37.804836251 +0000 UTC m=+19.888962671" watchObservedRunningTime="2026-10-30 19:29:37.805778644 +0000 UTC m=+19.889905062"
Oct 30 19:29:38.477591 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:38.477554    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:38.477739 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:38.477649    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wwsm" podUID="926e8fa4-1a22-48b1-832e-ea924ef61178"
Oct 30 19:29:38.477739 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:38.477669    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:38.477834 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:38.477750    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zdqg" podUID="bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e"
Oct 30 19:29:38.477834 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:38.477774    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:38.477936 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:38.477852    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b4p" podUID="bb3df8dc-d134-4cee-b95e-21e32f1df218"
Oct 30 19:29:38.784106 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:38.784074    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h" event={"ID":"c6ec634f-81f4-4d4e-97ef-33092b38052a","Type":"ContainerStarted","Data":"0484e9907bb287d705a5eb41c5f293c0c7babd871f88b232ed9d7495ab260a95"}
Oct 30 19:29:40.476593 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:40.476406    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:40.477207 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:40.476404    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:40.477207 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:40.476693    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b4p" podUID="bb3df8dc-d134-4cee-b95e-21e32f1df218"
Oct 30 19:29:40.477207 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:40.476433    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:40.477207 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:40.476746    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wwsm" podUID="926e8fa4-1a22-48b1-832e-ea924ef61178"
Oct 30 19:29:40.477207 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:40.476837    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zdqg" podUID="bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e"
Oct 30 19:29:40.790948 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:40.790823    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h" event={"ID":"c6ec634f-81f4-4d4e-97ef-33092b38052a","Type":"ContainerStarted","Data":"a4a81a06cce8b5fb2a316041b6f04402f45f35f4e1a4931673fadf5874d80853"}
Oct 30 19:29:40.791188 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:40.791169    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:40.791251 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:40.791240    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:40.807065 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:40.807035    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:40.822014 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:40.821961    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h" podStartSLOduration=6.719642233 podStartE2EDuration="21.821948444s" totalImagesPullingTime="15.102306211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:40.821402092 +0000 UTC m=+22.905528507" watchObservedRunningTime="2026-10-30 19:29:40.821948444 +0000 UTC m=+22.906074872"
Oct 30 19:29:41.794021 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:41.793988    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:41.809890 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:41.809851    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:29:42.150174 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:42.150089    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6wwsm"]
Oct 30 19:29:42.150296 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:42.150239    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:42.150377 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:42.150356    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wwsm" podUID="926e8fa4-1a22-48b1-832e-ea924ef61178"
Oct 30 19:29:42.153384 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:42.153356    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4zdqg"]
Oct 30 19:29:42.153506 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:42.153459    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:42.153561 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:42.153530    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zdqg" podUID="bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e"
Oct 30 19:29:42.156316 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:42.156291    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-87b4p"]
Oct 30 19:29:42.156440 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:42.156405    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:42.156526 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:42.156510    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b4p" podUID="bb3df8dc-d134-4cee-b95e-21e32f1df218"
Oct 30 19:29:43.187677 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:43.187637    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:29:43.188201 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:43.187794    2569 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:29:43.188259 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:43.188203    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:29:43.476066 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:43.475978    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:43.476223 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:43.476085    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zdqg" podUID="bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e"
Oct 30 19:29:43.476223 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:43.476113    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:43.476223 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:43.476159    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wwsm" podUID="926e8fa4-1a22-48b1-832e-ea924ef61178"
Oct 30 19:29:44.476715 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:44.476686    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:44.477176 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:44.476832    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b4p" podUID="bb3df8dc-d134-4cee-b95e-21e32f1df218"
Oct 30 19:29:45.476275 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:45.476236    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:45.476483 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:45.476255    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:45.476483 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:45.476353    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wwsm" podUID="926e8fa4-1a22-48b1-832e-ea924ef61178"
Oct 30 19:29:45.476483 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:45.476410    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zdqg" podUID="bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e"
Oct 30 19:29:46.476081 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:46.475991    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:46.476439 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:46.476137    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87b4p" podUID="bb3df8dc-d134-4cee-b95e-21e32f1df218"
Oct 30 19:29:47.476352 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:47.476315    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:47.476724 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:47.476315    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:47.476724 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:47.476431    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wwsm" podUID="926e8fa4-1a22-48b1-832e-ea924ef61178"
Oct 30 19:29:47.476724 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:47.476488    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zdqg" podUID="bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e"
Oct 30 19:29:48.173661 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.173635    2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeReady"
Oct 30 19:29:48.173801 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.173748    2569 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 19:29:48.209161 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.209121    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ntrs2"]
Oct 30 19:29:48.212260 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.212231    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qjqz6"]
Oct 30 19:29:48.212408 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.212364    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ntrs2"
Oct 30 19:29:48.216805 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.216778    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v9m7c"]
Oct 30 19:29:48.217337 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.217316    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qjqz6"
Oct 30 19:29:48.220368 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.220344    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7ff684fc5d-nksx9"]
Oct 30 19:29:48.220484 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.220464    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-v9m7c"
Oct 30 19:29:48.223350 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.223330    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"]
Oct 30 19:29:48.223465 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.223449    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.226300 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.226281    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6cb4fdcd5-zsxtw"]
Oct 30 19:29:48.226398 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.226377    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:29:48.228919 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.228898    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 19:29:48.228919 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.228922    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 19:29:48.229300 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.229280    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4vrds"]
Oct 30 19:29:48.229740 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.229398    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:48.232205 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.232188    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hv5j2"]
Oct 30 19:29:48.232319 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.232303    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.233263 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.233246    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 19:29:48.233263 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.233246    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 19:29:48.233369 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.233246    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 19:29:48.234325 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.234114    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p4b2c\""
Oct 30 19:29:48.235219 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.235202    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-bwt58"]
Oct 30 19:29:48.235365 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.235350    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hv5j2"
Oct 30 19:29:48.244394 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.243527    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"]
Oct 30 19:29:48.245024 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.244764    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:29:48.245024 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.244919    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 19:29:48.257305 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.257276    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"]
Oct 30 19:29:48.257471 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.257384    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:29:48.260355 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.260336    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5fdc8795f9-cfh7p"]
Oct 30 19:29:48.260478 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.260451    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:29:48.265829 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.265801    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"]
Oct 30 19:29:48.267928 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.267904    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.274970 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.274944    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"]
Oct 30 19:29:48.275152 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.275133    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:29:48.282260 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.282238    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-j6dm5"]
Oct 30 19:29:48.282390 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.282371    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:29:48.285575 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.285552    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ntrs2"]
Oct 30 19:29:48.286064 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.286043    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j6dm5"
Oct 30 19:29:48.286454 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.285597    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qjqz6"]
Oct 30 19:29:48.286533 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.286484    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kl9jq"]
Oct 30 19:29:48.292326 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.292290    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vzpfb"]
Oct 30 19:29:48.292593 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.292565    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:29:48.295659 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.295615    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"]
Oct 30 19:29:48.295659 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.295657    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hv5j2"]
Oct 30 19:29:48.295809 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.295675    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-j6dm5"]
Oct 30 19:29:48.295809 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.295689    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-bwt58"]
Oct 30 19:29:48.295809 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.295708    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"]
Oct 30 19:29:48.295809 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.295726    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"]
Oct 30 19:29:48.295809 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.295740    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7ff684fc5d-nksx9"]
Oct 30 19:29:48.295809 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.295754    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6cb4fdcd5-zsxtw"]
Oct 30 19:29:48.295809 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.295767    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fdc8795f9-cfh7p"]
Oct 30 19:29:48.295809 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.295780    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v9m7c"]
Oct 30 19:29:48.295809 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.295792    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kl9jq"]
Oct 30 19:29:48.295809 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.295808    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vzpfb"]
Oct 30 19:29:48.296101 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.295819    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"]
Oct 30 19:29:48.296101 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.295835    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4vrds"]
Oct 30 19:29:48.296101 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.295852    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"]
Oct 30 19:29:48.296101 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.295943    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vzpfb"
Oct 30 19:29:48.323258 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.323196    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:29:48.323258 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.323218    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:29:48.323447 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.323318    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:29:48.323447 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.323349    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:29:48.323633 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.323611    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 19:29:48.323800 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.323689    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 19:29:48.323800 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.323776    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:29:48.323973 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.323931    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:29:48.324037 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.323978    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 19:29:48.324037 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.324021    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 19:29:48.324140 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.324056    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:29:48.324140 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.324113    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 19:29:48.324276 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.324192    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:29:48.324592 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.324568    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 19:29:48.324678 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.324587    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 19:29:48.324678 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.324647    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:29:48.324781 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.324688    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:29:48.324781 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.324708    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:29:48.324781 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.324724    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:29:48.324781 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.324744    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 19:29:48.324781 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.324762    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:29:48.325089 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.324828    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 19:29:48.325089 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.324836    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 19:29:48.325089 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.324905    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:29:48.325089 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.325070    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:29:48.325089 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.325089    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:29:48.325373 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.325348    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 19:29:48.325597 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.325582    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:29:48.326346 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.326332    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 19:29:48.328518 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.328500    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:29:48.328766 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.328750    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 19:29:48.329998 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.329834    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 19:29:48.329998 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.329855    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 19:29:48.329998 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.329885    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 19:29:48.329998 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.329886    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 19:29:48.329998 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.329938    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:29:48.329998 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.329940    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 19:29:48.330459 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330109    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 19:29:48.330459 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330152    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:29:48.330459 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330181    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xgmwc\""
Oct 30 19:29:48.330459 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330184    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rhpjd\""
Oct 30 19:29:48.330459 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330195    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 19:29:48.330459 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330271    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 19:29:48.330459 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330276    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hpls5\""
Oct 30 19:29:48.330459 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330306    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 19:29:48.330459 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330331    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-psc6g\""
Oct 30 19:29:48.330459 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330334    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bkswr\""
Oct 30 19:29:48.330459 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330355    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-s4jds\""
Oct 30 19:29:48.330459 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330372    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 19:29:48.330459 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330432    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bvpxn\""
Oct 30 19:29:48.330459 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330460    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-svfmj\""
Oct 30 19:29:48.331138 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330462    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pk7z8\""
Oct 30 19:29:48.331138 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330485    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-8ztmt\""
Oct 30 19:29:48.331138 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330504    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ttn9c\""
Oct 30 19:29:48.331138 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.330506    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j9zzn\""
Oct 30 19:29:48.353343 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353315    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:29:48.353461 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/468ec5d6-34c5-4577-84a5-ec661a3f2736-serving-cert\") pod \"service-ca-operator-544d749c8c-d2pkv\" (UID: \"468ec5d6-34c5-4577-84a5-ec661a3f2736\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:29:48.353461 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353416    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/392e4212-393d-4aaa-904b-f72f1e287c34-config\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:29:48.353461 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353440    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb4cf\" (UniqueName: \"kubernetes.io/projected/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-kube-api-access-cb4cf\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:29:48.353461 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353458    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-image-registry-private-configuration\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.353619 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353480    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp8kb\" (UniqueName: \"kubernetes.io/projected/6bc162c8-806a-4d1f-a046-3ade90069067-kube-api-access-mp8kb\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.353619 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353532    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:29:48.353619 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353557    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e0ec109-980a-4203-b45b-25ebd01ae8ed-ca-trust-extracted\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.353619 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353594    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn4pl\" (UniqueName: \"kubernetes.io/projected/392e4212-393d-4aaa-904b-f72f1e287c34-kube-api-access-kn4pl\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:29:48.353798 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353653    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.353798 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353725    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-default-certificate\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:48.353798 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353763    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:48.353921 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353800    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-certificates\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.353921 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353834    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzjfz\" (UniqueName: \"kubernetes.io/projected/0af906a5-4f1e-4feb-8e1d-366cce9964c0-kube-api-access-rzjfz\") pod \"kas-connection-checker-685895c7c9-qjqz6\" (UID: \"0af906a5-4f1e-4feb-8e1d-366cce9964c0\") " pod="kube-system/kas-connection-checker-685895c7c9-qjqz6"
Oct 30 19:29:48.353921 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353890    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3615ac3b-837c-46ea-b575-4cdc1edc78a0-cert\") pod \"ingress-canary-vzpfb\" (UID: \"3615ac3b-837c-46ea-b575-4cdc1edc78a0\") " pod="openshift-ingress-canary/ingress-canary-vzpfb"
Oct 30 19:29:48.353921 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353907    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfdmm\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-kube-api-access-zfdmm\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.354094 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353922    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4909e20c-2484-41bd-8f06-67f37dfd337b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk\" (UID: \"4909e20c-2484-41bd-8f06-67f37dfd337b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:29:48.354094 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353944    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc162c8-806a-4d1f-a046-3ade90069067-service-ca-bundle\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.354094 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.353979    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/153c1fd1-90a2-4063-b066-6503d796d6d8-ca-trust-extracted\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.354094 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354014    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-trusted-ca\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.354094 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354059    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vxfk\" (UniqueName: \"kubernetes.io/projected/ac07aa06-d891-4c95-8500-fdef3723ff38-kube-api-access-5vxfk\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:29:48.354269 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354099    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-installation-pull-secrets\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.354269 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354119    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/392e4212-393d-4aaa-904b-f72f1e287c34-serving-cert\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:29:48.354269 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354135    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:29:48.354269 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354151    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bc162c8-806a-4d1f-a046-3ade90069067-tmp\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.354269 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354166    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:29:48.354269 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354218    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:29:48.354269 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354257    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-bound-sa-token\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.354538 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354291    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac07aa06-d891-4c95-8500-fdef3723ff38-tmp-dir\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:29:48.354538 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354309    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/392e4212-393d-4aaa-904b-f72f1e287c34-trusted-ca\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:29:48.354538 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354331    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtz66\" (UniqueName: \"kubernetes.io/projected/d4c7253d-c52b-4f85-bc06-cfafe6443bb4-kube-api-access-gtz66\") pod \"volume-data-source-validator-58c9f5f46d-hv5j2\" (UID: \"d4c7253d-c52b-4f85-bc06-cfafe6443bb4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hv5j2"
Oct 30 19:29:48.354538 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354360    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xd7c\" (UniqueName: \"kubernetes.io/projected/39126eba-223c-47c2-aff5-4233035260a3-kube-api-access-7xd7c\") pod \"kas-connection-checker-685895c7c9-ntrs2\" (UID: \"39126eba-223c-47c2-aff5-4233035260a3\") " pod="kube-system/kas-connection-checker-685895c7c9-ntrs2"
Oct 30 19:29:48.354538 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354392    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/468ec5d6-34c5-4577-84a5-ec661a3f2736-config\") pod \"service-ca-operator-544d749c8c-d2pkv\" (UID: \"468ec5d6-34c5-4577-84a5-ec661a3f2736\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:29:48.354538 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354463    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-image-registry-private-configuration\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.354538 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354496    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bwt58\" (UID: \"0c796373-b082-4069-b5e7-228588cc99da\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:29:48.354538 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354530    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-trusted-ca\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.354936 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354600    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjmx9\" (UniqueName: \"kubernetes.io/projected/4909e20c-2484-41bd-8f06-67f37dfd337b-kube-api-access-xjmx9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk\" (UID: \"4909e20c-2484-41bd-8f06-67f37dfd337b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:29:48.354936 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354633    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c796373-b082-4069-b5e7-228588cc99da-nginx-conf\") pod \"networking-console-plugin-86c478bb88-bwt58\" (UID: \"0c796373-b082-4069-b5e7-228588cc99da\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:29:48.354936 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354712    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6qvt\" (UniqueName: \"kubernetes.io/projected/468ec5d6-34c5-4577-84a5-ec661a3f2736-kube-api-access-h6qvt\") pod \"service-ca-operator-544d749c8c-d2pkv\" (UID: \"468ec5d6-34c5-4577-84a5-ec661a3f2736\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:29:48.354936 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354737    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q76f\" (UniqueName: \"kubernetes.io/projected/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-kube-api-access-5q76f\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:48.354936 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354770    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6bc162c8-806a-4d1f-a046-3ade90069067-snapshots\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.354936 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354800    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6bc162c8-806a-4d1f-a046-3ade90069067-serving-cert\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.354936 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354838    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac07aa06-d891-4c95-8500-fdef3723ff38-config-volume\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:29:48.354936 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354919    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.355244 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.354972    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhbv6\" (UniqueName: \"kubernetes.io/projected/3615ac3b-837c-46ea-b575-4cdc1edc78a0-kube-api-access-jhbv6\") pod \"ingress-canary-vzpfb\" (UID: \"3615ac3b-837c-46ea-b575-4cdc1edc78a0\") " pod="openshift-ingress-canary/ingress-canary-vzpfb"
Oct 30 19:29:48.355244 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.355022    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s4j8\" (UniqueName: \"kubernetes.io/projected/f387d9b5-577c-470c-a25e-da28234f464a-kube-api-access-8s4j8\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:29:48.355244 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.355047    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-stats-auth\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:48.355244 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.355065    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-certificates\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.355244 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.355080    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b8qf\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-kube-api-access-6b8qf\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.355244 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.355154    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc162c8-806a-4d1f-a046-3ade90069067-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.355244 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.355192    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:48.355244 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.355240    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-bound-sa-token\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.355591 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.355275    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-installation-pull-secrets\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.355591 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.355316    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-884tr\" (UniqueName: \"kubernetes.io/projected/7941072e-d482-42dc-a44e-6e0f9118912f-kube-api-access-884tr\") pod \"network-check-source-6f78d59f99-j6dm5\" (UID: \"7941072e-d482-42dc-a44e-6e0f9118912f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j6dm5"
Oct 30 19:29:48.355591 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.355356    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bbgk\" (UniqueName: \"kubernetes.io/projected/28217977-99fe-4540-a322-3b3efb6dfa36-kube-api-access-2bbgk\") pod \"kas-connection-checker-685895c7c9-v9m7c\" (UID: \"28217977-99fe-4540-a322-3b3efb6dfa36\") " pod="kube-system/kas-connection-checker-685895c7c9-v9m7c"
Oct 30 19:29:48.355591 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.355381    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4909e20c-2484-41bd-8f06-67f37dfd337b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk\" (UID: \"4909e20c-2484-41bd-8f06-67f37dfd337b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:29:48.456312 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456057    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjmx9\" (UniqueName: \"kubernetes.io/projected/4909e20c-2484-41bd-8f06-67f37dfd337b-kube-api-access-xjmx9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk\" (UID: \"4909e20c-2484-41bd-8f06-67f37dfd337b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:29:48.456482 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456336    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c796373-b082-4069-b5e7-228588cc99da-nginx-conf\") pod \"networking-console-plugin-86c478bb88-bwt58\" (UID: \"0c796373-b082-4069-b5e7-228588cc99da\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:29:48.456482 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456367    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6qvt\" (UniqueName: \"kubernetes.io/projected/468ec5d6-34c5-4577-84a5-ec661a3f2736-kube-api-access-h6qvt\") pod \"service-ca-operator-544d749c8c-d2pkv\" (UID: \"468ec5d6-34c5-4577-84a5-ec661a3f2736\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:29:48.456482 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456389    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q76f\" (UniqueName: \"kubernetes.io/projected/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-kube-api-access-5q76f\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:48.456482 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456428    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6bc162c8-806a-4d1f-a046-3ade90069067-snapshots\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.456482 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456449    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6bc162c8-806a-4d1f-a046-3ade90069067-serving-cert\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.456482 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456474    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac07aa06-d891-4c95-8500-fdef3723ff38-config-volume\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:29:48.456760 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456513    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.456760 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456538    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhbv6\" (UniqueName: \"kubernetes.io/projected/3615ac3b-837c-46ea-b575-4cdc1edc78a0-kube-api-access-jhbv6\") pod \"ingress-canary-vzpfb\" (UID: \"3615ac3b-837c-46ea-b575-4cdc1edc78a0\") " pod="openshift-ingress-canary/ingress-canary-vzpfb"
Oct 30 19:29:48.456760 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456562    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s4j8\" (UniqueName: \"kubernetes.io/projected/f387d9b5-577c-470c-a25e-da28234f464a-kube-api-access-8s4j8\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:29:48.456760 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456587    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-stats-auth\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:48.456760 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456610    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-certificates\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.456760 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456632    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b8qf\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-kube-api-access-6b8qf\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.456760 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456658    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc162c8-806a-4d1f-a046-3ade90069067-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.456760 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456680    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:48.456760 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.456689    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:29:48.456760 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456703    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-bound-sa-token\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.456760 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.456709    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7ff684fc5d-nksx9: secret "image-registry-tls" not found
Oct 30 19:29:48.456760 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456726    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-installation-pull-secrets\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.456760 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456754    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-884tr\" (UniqueName: \"kubernetes.io/projected/7941072e-d482-42dc-a44e-6e0f9118912f-kube-api-access-884tr\") pod \"network-check-source-6f78d59f99-j6dm5\" (UID: \"7941072e-d482-42dc-a44e-6e0f9118912f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j6dm5"
Oct 30 19:29:48.456760 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.456776    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls podName:9e0ec109-980a-4203-b45b-25ebd01ae8ed nodeName:}" failed. No retries permitted until 2026-10-30 19:29:48.956756661 +0000 UTC m=+31.040883057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls") pod "image-registry-7ff684fc5d-nksx9" (UID: "9e0ec109-980a-4203-b45b-25ebd01ae8ed") : secret "image-registry-tls" not found
Oct 30 19:29:48.458136 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456816    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bbgk\" (UniqueName: \"kubernetes.io/projected/28217977-99fe-4540-a322-3b3efb6dfa36-kube-api-access-2bbgk\") pod \"kas-connection-checker-685895c7c9-v9m7c\" (UID: \"28217977-99fe-4540-a322-3b3efb6dfa36\") " pod="kube-system/kas-connection-checker-685895c7c9-v9m7c"
Oct 30 19:29:48.458136 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456852    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4909e20c-2484-41bd-8f06-67f37dfd337b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk\" (UID: \"4909e20c-2484-41bd-8f06-67f37dfd337b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:29:48.458136 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456907    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:29:48.458136 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456936    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/468ec5d6-34c5-4577-84a5-ec661a3f2736-serving-cert\") pod \"service-ca-operator-544d749c8c-d2pkv\" (UID: \"468ec5d6-34c5-4577-84a5-ec661a3f2736\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:29:48.458136 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.456980    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/392e4212-393d-4aaa-904b-f72f1e287c34-config\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:29:48.458136 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457008    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb4cf\" (UniqueName: \"kubernetes.io/projected/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-kube-api-access-cb4cf\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:29:48.458136 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457036    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-image-registry-private-configuration\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.458136 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457067    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp8kb\" (UniqueName: \"kubernetes.io/projected/6bc162c8-806a-4d1f-a046-3ade90069067-kube-api-access-mp8kb\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.458136 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457090    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:29:48.458136 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457117    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e0ec109-980a-4203-b45b-25ebd01ae8ed-ca-trust-extracted\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.458136 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457147    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn4pl\" (UniqueName: \"kubernetes.io/projected/392e4212-393d-4aaa-904b-f72f1e287c34-kube-api-access-kn4pl\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:29:48.458136 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457183    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.458136 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457210    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c796373-b082-4069-b5e7-228588cc99da-nginx-conf\") pod \"networking-console-plugin-86c478bb88-bwt58\" (UID: \"0c796373-b082-4069-b5e7-228588cc99da\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:29:48.458136 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457214    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-default-certificate\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:48.458136 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457262    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:48.458884 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457290    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-certificates\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.458884 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457323    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzjfz\" (UniqueName: \"kubernetes.io/projected/0af906a5-4f1e-4feb-8e1d-366cce9964c0-kube-api-access-rzjfz\") pod \"kas-connection-checker-685895c7c9-qjqz6\" (UID: \"0af906a5-4f1e-4feb-8e1d-366cce9964c0\") " pod="kube-system/kas-connection-checker-685895c7c9-qjqz6"
Oct 30 19:29:48.458884 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457347    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3615ac3b-837c-46ea-b575-4cdc1edc78a0-cert\") pod \"ingress-canary-vzpfb\" (UID: \"3615ac3b-837c-46ea-b575-4cdc1edc78a0\") " pod="openshift-ingress-canary/ingress-canary-vzpfb"
Oct 30 19:29:48.458884 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457392    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfdmm\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-kube-api-access-zfdmm\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.458884 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.457437    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle podName:df3b6a17-4bbf-41dc-83e5-df5172b87b7f nodeName:}" failed. No retries permitted until 2026-10-30 19:29:48.957418145 +0000 UTC m=+31.041544553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle") pod "router-default-6cb4fdcd5-zsxtw" (UID: "df3b6a17-4bbf-41dc-83e5-df5172b87b7f") : configmap references non-existent config key: service-ca.crt
Oct 30 19:29:48.458884 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457451    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4909e20c-2484-41bd-8f06-67f37dfd337b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk\" (UID: \"4909e20c-2484-41bd-8f06-67f37dfd337b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:29:48.458884 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457482    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc162c8-806a-4d1f-a046-3ade90069067-service-ca-bundle\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.458884 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457504    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/153c1fd1-90a2-4063-b066-6503d796d6d8-ca-trust-extracted\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.458884 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457525    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-trusted-ca\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.458884 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457549    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vxfk\" (UniqueName: \"kubernetes.io/projected/ac07aa06-d891-4c95-8500-fdef3723ff38-kube-api-access-5vxfk\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:29:48.458884 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457569    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-installation-pull-secrets\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.458884 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457596    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/392e4212-393d-4aaa-904b-f72f1e287c34-serving-cert\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:29:48.458884 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457620    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:29:48.458884 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457645    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bc162c8-806a-4d1f-a046-3ade90069067-tmp\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.458884 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457666    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:29:48.459687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457686    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:29:48.459687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457710    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-bound-sa-token\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.459687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457736    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac07aa06-d891-4c95-8500-fdef3723ff38-tmp-dir\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:29:48.459687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457758    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/392e4212-393d-4aaa-904b-f72f1e287c34-trusted-ca\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:29:48.459687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457780    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtz66\" (UniqueName: \"kubernetes.io/projected/d4c7253d-c52b-4f85-bc06-cfafe6443bb4-kube-api-access-gtz66\") pod \"volume-data-source-validator-58c9f5f46d-hv5j2\" (UID: \"d4c7253d-c52b-4f85-bc06-cfafe6443bb4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hv5j2"
Oct 30 19:29:48.459687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457803    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xd7c\" (UniqueName: \"kubernetes.io/projected/39126eba-223c-47c2-aff5-4233035260a3-kube-api-access-7xd7c\") pod \"kas-connection-checker-685895c7c9-ntrs2\" (UID: \"39126eba-223c-47c2-aff5-4233035260a3\") " pod="kube-system/kas-connection-checker-685895c7c9-ntrs2"
Oct 30 19:29:48.459687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457827    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/468ec5d6-34c5-4577-84a5-ec661a3f2736-config\") pod \"service-ca-operator-544d749c8c-d2pkv\" (UID: \"468ec5d6-34c5-4577-84a5-ec661a3f2736\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:29:48.459687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457854    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-image-registry-private-configuration\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.459687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457900    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bwt58\" (UID: \"0c796373-b082-4069-b5e7-228588cc99da\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:29:48.459687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457926    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-trusted-ca\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.459687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.457955    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-certificates\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.459687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.458208    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4909e20c-2484-41bd-8f06-67f37dfd337b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk\" (UID: \"4909e20c-2484-41bd-8f06-67f37dfd337b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:29:48.459687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.458504    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac07aa06-d891-4c95-8500-fdef3723ff38-config-volume\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:29:48.459687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.458675    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-certificates\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.459687 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.458929    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6bc162c8-806a-4d1f-a046-3ade90069067-snapshots\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.460502 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.459702    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-trusted-ca\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.460502 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.459921    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc162c8-806a-4d1f-a046-3ade90069067-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.460502 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.460267    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/392e4212-393d-4aaa-904b-f72f1e287c34-trusted-ca\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:29:48.460502 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.460357    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc162c8-806a-4d1f-a046-3ade90069067-service-ca-bundle\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.460502 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.460437    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:29:48.460502 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.460499    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs podName:df3b6a17-4bbf-41dc-83e5-df5172b87b7f nodeName:}" failed. No retries permitted until 2026-10-30 19:29:48.960482376 +0000 UTC m=+31.044608785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs") pod "router-default-6cb4fdcd5-zsxtw" (UID: "df3b6a17-4bbf-41dc-83e5-df5172b87b7f") : secret "router-metrics-certs-default" not found
Oct 30 19:29:48.461132 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.460573    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac07aa06-d891-4c95-8500-fdef3723ff38-tmp-dir\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:29:48.461132 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.460720    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:29:48.461132 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.460734    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:29:48.461132 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.460783    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls podName:7427b4f7-fba8-4cde-ad90-1f46d5c92d87 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:48.960767078 +0000 UTC m=+31.044893472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hlvg8" (UID: "7427b4f7-fba8-4cde-ad90-1f46d5c92d87") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:29:48.461132 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.460791    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/153c1fd1-90a2-4063-b066-6503d796d6d8-ca-trust-extracted\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.462466 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.462439    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4909e20c-2484-41bd-8f06-67f37dfd337b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk\" (UID: \"4909e20c-2484-41bd-8f06-67f37dfd337b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:29:48.462572 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.462547    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3615ac3b-837c-46ea-b575-4cdc1edc78a0-cert\") pod \"ingress-canary-vzpfb\" (UID: \"3615ac3b-837c-46ea-b575-4cdc1edc78a0\") " pod="openshift-ingress-canary/ingress-canary-vzpfb"
Oct 30 19:29:48.462656 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.462638    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-default-certificate\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:48.462769 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.462747    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:29:48.462923 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.462790    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e0ec109-980a-4203-b45b-25ebd01ae8ed-ca-trust-extracted\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.462923 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.462808    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert podName:0c796373-b082-4069-b5e7-228588cc99da nodeName:}" failed. No retries permitted until 2026-10-30 19:29:48.962792534 +0000 UTC m=+31.046918929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bwt58" (UID: "0c796373-b082-4069-b5e7-228588cc99da") : secret "networking-console-plugin-cert" not found
Oct 30 19:29:48.462923 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.462913    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:29:48.463115 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.462929    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fdc8795f9-cfh7p: secret "image-registry-tls" not found
Oct 30 19:29:48.463115 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.462971    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls podName:153c1fd1-90a2-4063-b066-6503d796d6d8 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:48.962960325 +0000 UTC m=+31.047086732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls") pod "image-registry-5fdc8795f9-cfh7p" (UID: "153c1fd1-90a2-4063-b066-6503d796d6d8") : secret "image-registry-tls" not found
Oct 30 19:29:48.463529 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.463509    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:29:48.463594 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.463581    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls podName:ac07aa06-d891-4c95-8500-fdef3723ff38 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:48.963564267 +0000 UTC m=+31.047690675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls") pod "dns-default-kl9jq" (UID: "ac07aa06-d891-4c95-8500-fdef3723ff38") : secret "dns-default-metrics-tls" not found
Oct 30 19:29:48.463979 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.463960    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/392e4212-393d-4aaa-904b-f72f1e287c34-config\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:29:48.464400 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.464375    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-trusted-ca\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.464785 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.464759    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bc162c8-806a-4d1f-a046-3ade90069067-tmp\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.465118 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.465093    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-image-registry-private-configuration\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.465264 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.465244    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/468ec5d6-34c5-4577-84a5-ec661a3f2736-config\") pod \"service-ca-operator-544d749c8c-d2pkv\" (UID: \"468ec5d6-34c5-4577-84a5-ec661a3f2736\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:29:48.465402 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.465377    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-stats-auth\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:48.465470 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.465450    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6bc162c8-806a-4d1f-a046-3ade90069067-serving-cert\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.465591 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.465574    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:29:48.465662 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.465642    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls podName:f387d9b5-577c-470c-a25e-da28234f464a nodeName:}" failed. No retries permitted until 2026-10-30 19:29:48.965625811 +0000 UTC m=+31.049752220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xdndw" (UID: "f387d9b5-577c-470c-a25e-da28234f464a") : secret "samples-operator-tls" not found
Oct 30 19:29:48.466074 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.466036    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-installation-pull-secrets\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.466212 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.466190    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:29:48.466923 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.466846    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/468ec5d6-34c5-4577-84a5-ec661a3f2736-serving-cert\") pod \"service-ca-operator-544d749c8c-d2pkv\" (UID: \"468ec5d6-34c5-4577-84a5-ec661a3f2736\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:29:48.466977 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.466883    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-installation-pull-secrets\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.467536 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.467520    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/392e4212-393d-4aaa-904b-f72f1e287c34-serving-cert\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:29:48.467815 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.467792    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-image-registry-private-configuration\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.471725 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.471696    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q76f\" (UniqueName: \"kubernetes.io/projected/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-kube-api-access-5q76f\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:48.471830 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.471757    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6qvt\" (UniqueName: \"kubernetes.io/projected/468ec5d6-34c5-4577-84a5-ec661a3f2736-kube-api-access-h6qvt\") pod \"service-ca-operator-544d749c8c-d2pkv\" (UID: \"468ec5d6-34c5-4577-84a5-ec661a3f2736\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:29:48.472003 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.471980    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjmx9\" (UniqueName: \"kubernetes.io/projected/4909e20c-2484-41bd-8f06-67f37dfd337b-kube-api-access-xjmx9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk\" (UID: \"4909e20c-2484-41bd-8f06-67f37dfd337b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:29:48.472071 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.472037    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtz66\" (UniqueName: \"kubernetes.io/projected/d4c7253d-c52b-4f85-bc06-cfafe6443bb4-kube-api-access-gtz66\") pod \"volume-data-source-validator-58c9f5f46d-hv5j2\" (UID: \"d4c7253d-c52b-4f85-bc06-cfafe6443bb4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hv5j2"
Oct 30 19:29:48.473011 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.472983    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s4j8\" (UniqueName: \"kubernetes.io/projected/f387d9b5-577c-470c-a25e-da28234f464a-kube-api-access-8s4j8\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:29:48.476380 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.476356    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:48.479104 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.479081    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bbgk\" (UniqueName: \"kubernetes.io/projected/28217977-99fe-4540-a322-3b3efb6dfa36-kube-api-access-2bbgk\") pod \"kas-connection-checker-685895c7c9-v9m7c\" (UID: \"28217977-99fe-4540-a322-3b3efb6dfa36\") " pod="kube-system/kas-connection-checker-685895c7c9-v9m7c"
Oct 30 19:29:48.479315 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.479284    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xd7c\" (UniqueName: \"kubernetes.io/projected/39126eba-223c-47c2-aff5-4233035260a3-kube-api-access-7xd7c\") pod \"kas-connection-checker-685895c7c9-ntrs2\" (UID: \"39126eba-223c-47c2-aff5-4233035260a3\") " pod="kube-system/kas-connection-checker-685895c7c9-ntrs2"
Oct 30 19:29:48.479374 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.479316    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-bound-sa-token\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.479534 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.479427    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhbv6\" (UniqueName: \"kubernetes.io/projected/3615ac3b-837c-46ea-b575-4cdc1edc78a0-kube-api-access-jhbv6\") pod \"ingress-canary-vzpfb\" (UID: \"3615ac3b-837c-46ea-b575-4cdc1edc78a0\") " pod="openshift-ingress-canary/ingress-canary-vzpfb"
Oct 30 19:29:48.479843 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.479818    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb4cf\" (UniqueName: \"kubernetes.io/projected/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-kube-api-access-cb4cf\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:29:48.480729 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.480689    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:29:48.480837 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.480743    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8hwq2\""
Oct 30 19:29:48.481984 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.481961    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-884tr\" (UniqueName: \"kubernetes.io/projected/7941072e-d482-42dc-a44e-6e0f9118912f-kube-api-access-884tr\") pod \"network-check-source-6f78d59f99-j6dm5\" (UID: \"7941072e-d482-42dc-a44e-6e0f9118912f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j6dm5"
Oct 30 19:29:48.482646 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.482625    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b8qf\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-kube-api-access-6b8qf\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.488848 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.488820    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vxfk\" (UniqueName: \"kubernetes.io/projected/ac07aa06-d891-4c95-8500-fdef3723ff38-kube-api-access-5vxfk\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:29:48.489054 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.489034    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp8kb\" (UniqueName: \"kubernetes.io/projected/6bc162c8-806a-4d1f-a046-3ade90069067-kube-api-access-mp8kb\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.489247 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.489228    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn4pl\" (UniqueName: \"kubernetes.io/projected/392e4212-393d-4aaa-904b-f72f1e287c34-kube-api-access-kn4pl\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:29:48.489845 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.489830    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzjfz\" (UniqueName: \"kubernetes.io/projected/0af906a5-4f1e-4feb-8e1d-366cce9964c0-kube-api-access-rzjfz\") pod \"kas-connection-checker-685895c7c9-qjqz6\" (UID: \"0af906a5-4f1e-4feb-8e1d-366cce9964c0\") " pod="kube-system/kas-connection-checker-685895c7c9-qjqz6"
Oct 30 19:29:48.493716 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.493693    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-bound-sa-token\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.499195 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.499170    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfdmm\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-kube-api-access-zfdmm\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.527900 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.527854    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ntrs2"
Oct 30 19:29:48.534994 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.534970    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qjqz6"
Oct 30 19:29:48.556083 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.556052    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-v9m7c"
Oct 30 19:29:48.570648 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.570601    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:29:48.583980 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.583804    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:29:48.595038 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.595008    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hv5j2"
Oct 30 19:29:48.619650 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.619368    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:29:48.646530 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.646131    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j6dm5"
Oct 30 19:29:48.649318 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.647205    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:29:48.656503 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.656432    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vzpfb"
Oct 30 19:29:48.832743 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.832672    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qjqz6"]
Oct 30 19:29:48.835499 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.835475    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ntrs2"]
Oct 30 19:29:48.836632 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:48.836610    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af906a5_4f1e_4feb_8e1d_366cce9964c0.slice/crio-234270c9f293615bc63d23ff406cb11dc35111515be70d1f31c5a069a2e9f8be WatchSource:0}: Error finding container 234270c9f293615bc63d23ff406cb11dc35111515be70d1f31c5a069a2e9f8be: Status 404 returned error can't find the container with id 234270c9f293615bc63d23ff406cb11dc35111515be70d1f31c5a069a2e9f8be
Oct 30 19:29:48.838506 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:48.838481    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39126eba_223c_47c2_aff5_4233035260a3.slice/crio-de4b10692573ed031d00c1d6fcb29945422022be92d25e9d2e6f7c4be83948b9 WatchSource:0}: Error finding container de4b10692573ed031d00c1d6fcb29945422022be92d25e9d2e6f7c4be83948b9: Status 404 returned error can't find the container with id de4b10692573ed031d00c1d6fcb29945422022be92d25e9d2e6f7c4be83948b9
Oct 30 19:29:48.864189 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.864111    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v9m7c"]
Oct 30 19:29:48.871598 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:48.871519    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28217977_99fe_4540_a322_3b3efb6dfa36.slice/crio-159873945b00a7e58f88326c67234600440fdacd52e6967ad11414cc0039e831 WatchSource:0}: Error finding container 159873945b00a7e58f88326c67234600440fdacd52e6967ad11414cc0039e831: Status 404 returned error can't find the container with id 159873945b00a7e58f88326c67234600440fdacd52e6967ad11414cc0039e831
Oct 30 19:29:48.886393 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.886352    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"]
Oct 30 19:29:48.899849 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:48.899809    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4909e20c_2484_41bd_8f06_67f37dfd337b.slice/crio-e7bdcfe753eeaad93d91080684857987cc8d9e8813f718e927802a0681d5902b WatchSource:0}: Error finding container e7bdcfe753eeaad93d91080684857987cc8d9e8813f718e927802a0681d5902b: Status 404 returned error can't find the container with id e7bdcfe753eeaad93d91080684857987cc8d9e8813f718e927802a0681d5902b
Oct 30 19:29:48.908548 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.908474    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4vrds"]
Oct 30 19:29:48.913228 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.913007    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hv5j2"]
Oct 30 19:29:48.965932 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.965835    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bwt58\" (UID: \"0c796373-b082-4069-b5e7-228588cc99da\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:29:48.966595 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.966243    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:29:48.966595 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.966330    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert podName:0c796373-b082-4069-b5e7-228588cc99da nodeName:}" failed. No retries permitted until 2026-10-30 19:29:49.966309616 +0000 UTC m=+32.050436010 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bwt58" (UID: "0c796373-b082-4069-b5e7-228588cc99da") : secret "networking-console-plugin-cert" not found
Oct 30 19:29:48.967012 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.966912    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:29:48.967012 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.966932    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7ff684fc5d-nksx9: secret "image-registry-tls" not found
Oct 30 19:29:48.967012 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.966988    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls podName:9e0ec109-980a-4203-b45b-25ebd01ae8ed nodeName:}" failed. No retries permitted until 2026-10-30 19:29:49.96697117 +0000 UTC m=+32.051097568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls") pod "image-registry-7ff684fc5d-nksx9" (UID: "9e0ec109-980a-4203-b45b-25ebd01ae8ed") : secret "image-registry-tls" not found
Oct 30 19:29:48.968158 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.967243    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:48.968158 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.967309    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:48.968158 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.967391    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:48.968158 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.967422    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:48.968158 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.967472    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:29:48.968158 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.967502    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:29:48.968158 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.967528    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:29:48.968158 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.967651    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:29:48.968158 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.967697    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls podName:ac07aa06-d891-4c95-8500-fdef3723ff38 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:49.96768281 +0000 UTC m=+32.051809203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls") pod "dns-default-kl9jq" (UID: "ac07aa06-d891-4c95-8500-fdef3723ff38") : secret "dns-default-metrics-tls" not found
Oct 30 19:29:48.968158 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.967770    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:29:48.968158 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.967802    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs podName:df3b6a17-4bbf-41dc-83e5-df5172b87b7f nodeName:}" failed. No retries permitted until 2026-10-30 19:29:49.9677927 +0000 UTC m=+32.051919094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs") pod "router-default-6cb4fdcd5-zsxtw" (UID: "df3b6a17-4bbf-41dc-83e5-df5172b87b7f") : secret "router-metrics-certs-default" not found
Oct 30 19:29:48.968158 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.967884    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:29:48.968158 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.967895    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fdc8795f9-cfh7p: secret "image-registry-tls" not found
Oct 30 19:29:48.968158 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.967925    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls podName:153c1fd1-90a2-4063-b066-6503d796d6d8 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:49.967915399 +0000 UTC m=+32.052041796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls") pod "image-registry-5fdc8795f9-cfh7p" (UID: "153c1fd1-90a2-4063-b066-6503d796d6d8") : secret "image-registry-tls" not found
Oct 30 19:29:48.968158 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.967989    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle podName:df3b6a17-4bbf-41dc-83e5-df5172b87b7f nodeName:}" failed. No retries permitted until 2026-10-30 19:29:49.967980453 +0000 UTC m=+32.052106848 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle") pod "router-default-6cb4fdcd5-zsxtw" (UID: "df3b6a17-4bbf-41dc-83e5-df5172b87b7f") : configmap references non-existent config key: service-ca.crt
Oct 30 19:29:48.968158 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.968049    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:29:48.969059 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.968080    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls podName:7427b4f7-fba8-4cde-ad90-1f46d5c92d87 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:49.968071103 +0000 UTC m=+32.052197496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hlvg8" (UID: "7427b4f7-fba8-4cde-ad90-1f46d5c92d87") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:29:48.969226 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.969147    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:29:48.969226 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:48.969207    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls podName:f387d9b5-577c-470c-a25e-da28234f464a nodeName:}" failed. No retries permitted until 2026-10-30 19:29:49.969191541 +0000 UTC m=+32.053317937 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xdndw" (UID: "f387d9b5-577c-470c-a25e-da28234f464a") : secret "samples-operator-tls" not found
Oct 30 19:29:48.975375 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.975345    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"]
Oct 30 19:29:48.978235 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.978209    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"]
Oct 30 19:29:48.979380 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:48.979348    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468ec5d6_34c5_4577_84a5_ec661a3f2736.slice/crio-0b859724c833e248f69a3b21c3a0ed8a9270ce2a8e686d446d0856ad8f0392a2 WatchSource:0}: Error finding container 0b859724c833e248f69a3b21c3a0ed8a9270ce2a8e686d446d0856ad8f0392a2: Status 404 returned error can't find the container with id 0b859724c833e248f69a3b21c3a0ed8a9270ce2a8e686d446d0856ad8f0392a2
Oct 30 19:29:48.985928 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:48.985901    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392e4212_393d_4aaa_904b_f72f1e287c34.slice/crio-c8b94627ee953e7c239fc66fbf71993f6c710c9eacdb2763ca70ef6aaefc809f WatchSource:0}: Error finding container c8b94627ee953e7c239fc66fbf71993f6c710c9eacdb2763ca70ef6aaefc809f: Status 404 returned error can't find the container with id c8b94627ee953e7c239fc66fbf71993f6c710c9eacdb2763ca70ef6aaefc809f
Oct 30 19:29:48.993570 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:48.993497    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vzpfb"]
Oct 30 19:29:49.015309 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.015279    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-j6dm5"]
Oct 30 19:29:49.019656 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:49.019631    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7941072e_d482_42dc_a44e_6e0f9118912f.slice/crio-12b4dd3d73c6a1df2d18d4e074867d79373af5546eafec150feaf4aa45bad2ab WatchSource:0}: Error finding container 12b4dd3d73c6a1df2d18d4e074867d79373af5546eafec150feaf4aa45bad2ab: Status 404 returned error can't find the container with id 12b4dd3d73c6a1df2d18d4e074867d79373af5546eafec150feaf4aa45bad2ab
Oct 30 19:29:49.477042 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.477005    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:49.478107 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.477474    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:49.481154 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.481120    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-q9kwd\""
Oct 30 19:29:49.481349 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.481241    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:29:49.481349 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.481301    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2prmp\""
Oct 30 19:29:49.818337 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.818268    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4vrds" event={"ID":"6bc162c8-806a-4d1f-a046-3ade90069067","Type":"ContainerStarted","Data":"24eb387a935fbe718617009ea1258f85c1430143dda2b1b9ac47b8f4e16d4e35"}
Oct 30 19:29:49.822226 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.822133    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-v9m7c" event={"ID":"28217977-99fe-4540-a322-3b3efb6dfa36","Type":"ContainerStarted","Data":"6d4b60b14a069f61c854d4a95aa31c2407ed131ddfb5f6173d9b015fd415393c"}
Oct 30 19:29:49.822226 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.822192    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-v9m7c" event={"ID":"28217977-99fe-4540-a322-3b3efb6dfa36","Type":"ContainerStarted","Data":"159873945b00a7e58f88326c67234600440fdacd52e6967ad11414cc0039e831"}
Oct 30 19:29:49.828288 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.827721    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ntrs2" event={"ID":"39126eba-223c-47c2-aff5-4233035260a3","Type":"ContainerStarted","Data":"c507b3f3c4f44beb69050937eaa7e5f3d88bfef75d20f40bf49a691653cfbaeb"}
Oct 30 19:29:49.828288 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.827776    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ntrs2" event={"ID":"39126eba-223c-47c2-aff5-4233035260a3","Type":"ContainerStarted","Data":"de4b10692573ed031d00c1d6fcb29945422022be92d25e9d2e6f7c4be83948b9"}
Oct 30 19:29:49.830480 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.830450    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qjqz6" event={"ID":"0af906a5-4f1e-4feb-8e1d-366cce9964c0","Type":"ContainerStarted","Data":"365ca388fd20d41c81f261fc3ca4705ca19ae4f6800afba9a69f90fc4f06c9b3"}
Oct 30 19:29:49.830617 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.830499    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qjqz6" event={"ID":"0af906a5-4f1e-4feb-8e1d-366cce9964c0","Type":"ContainerStarted","Data":"234270c9f293615bc63d23ff406cb11dc35111515be70d1f31c5a069a2e9f8be"}
Oct 30 19:29:49.833127 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.833090    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hv5j2" event={"ID":"d4c7253d-c52b-4f85-bc06-cfafe6443bb4","Type":"ContainerStarted","Data":"bf58f46fc376b90dd9b3c55c60be823fca4491f1fe37b32845f586a907dbac35"}
Oct 30 19:29:49.834678 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.834614    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk" event={"ID":"4909e20c-2484-41bd-8f06-67f37dfd337b","Type":"ContainerStarted","Data":"e7bdcfe753eeaad93d91080684857987cc8d9e8813f718e927802a0681d5902b"}
Oct 30 19:29:49.843003 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.840665    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-v9m7c" podStartSLOduration=410.840648703 podStartE2EDuration="6m50.840648703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:49.839871341 +0000 UTC m=+31.923997748" watchObservedRunningTime="2026-10-30 19:29:49.840648703 +0000 UTC m=+31.924775121"
Oct 30 19:29:49.843525 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.843422    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vzpfb" event={"ID":"3615ac3b-837c-46ea-b575-4cdc1edc78a0","Type":"ContainerStarted","Data":"8a9e7044a68634c91220db20574d26df0b9c05fb61bed3060d0df1effb170ace"}
Oct 30 19:29:49.860354 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.860084    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx" event={"ID":"392e4212-393d-4aaa-904b-f72f1e287c34","Type":"ContainerStarted","Data":"c8b94627ee953e7c239fc66fbf71993f6c710c9eacdb2763ca70ef6aaefc809f"}
Oct 30 19:29:49.864962 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.864897    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ntrs2" podStartSLOduration=410.864878601 podStartE2EDuration="6m50.864878601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:49.864135802 +0000 UTC m=+31.948262221" watchObservedRunningTime="2026-10-30 19:29:49.864878601 +0000 UTC m=+31.949005015"
Oct 30 19:29:49.874547 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.874503    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j6dm5" event={"ID":"7941072e-d482-42dc-a44e-6e0f9118912f","Type":"ContainerStarted","Data":"12b4dd3d73c6a1df2d18d4e074867d79373af5546eafec150feaf4aa45bad2ab"}
Oct 30 19:29:49.876550 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.876490    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv" event={"ID":"468ec5d6-34c5-4577-84a5-ec661a3f2736","Type":"ContainerStarted","Data":"0b859724c833e248f69a3b21c3a0ed8a9270ce2a8e686d446d0856ad8f0392a2"}
Oct 30 19:29:49.980470 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.978286    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bwt58\" (UID: \"0c796373-b082-4069-b5e7-228588cc99da\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:29:49.980470 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.978381    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:49.980470 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.978422    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:49.980470 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.978507    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:49.980470 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.978536    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:49.980470 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.978596    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:29:49.980470 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.978626    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:29:49.980470 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:49.978650    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:29:49.980470 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.978791    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:29:49.980470 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.978852    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls podName:ac07aa06-d891-4c95-8500-fdef3723ff38 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:51.978834611 +0000 UTC m=+34.062961010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls") pod "dns-default-kl9jq" (UID: "ac07aa06-d891-4c95-8500-fdef3723ff38") : secret "dns-default-metrics-tls" not found
Oct 30 19:29:49.980470 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.979659    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:29:49.980470 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.979710    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert podName:0c796373-b082-4069-b5e7-228588cc99da nodeName:}" failed. No retries permitted until 2026-10-30 19:29:51.979694041 +0000 UTC m=+34.063820439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bwt58" (UID: "0c796373-b082-4069-b5e7-228588cc99da") : secret "networking-console-plugin-cert" not found
Oct 30 19:29:49.980470 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.979782    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:29:49.980470 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.979793    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7ff684fc5d-nksx9: secret "image-registry-tls" not found
Oct 30 19:29:49.980470 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.979824    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls podName:9e0ec109-980a-4203-b45b-25ebd01ae8ed nodeName:}" failed. No retries permitted until 2026-10-30 19:29:51.979813605 +0000 UTC m=+34.063940002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls") pod "image-registry-7ff684fc5d-nksx9" (UID: "9e0ec109-980a-4203-b45b-25ebd01ae8ed") : secret "image-registry-tls" not found
Oct 30 19:29:49.980470 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.979902    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:29:49.981391 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.979934    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs podName:df3b6a17-4bbf-41dc-83e5-df5172b87b7f nodeName:}" failed. No retries permitted until 2026-10-30 19:29:51.979923265 +0000 UTC m=+34.064049664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs") pod "router-default-6cb4fdcd5-zsxtw" (UID: "df3b6a17-4bbf-41dc-83e5-df5172b87b7f") : secret "router-metrics-certs-default" not found
Oct 30 19:29:49.981391 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.979991    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:29:49.981391 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.980010    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:29:49.981391 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.980023    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls podName:7427b4f7-fba8-4cde-ad90-1f46d5c92d87 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:51.980013462 +0000 UTC m=+34.064139857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hlvg8" (UID: "7427b4f7-fba8-4cde-ad90-1f46d5c92d87") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:29:49.981391 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.980025    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fdc8795f9-cfh7p: secret "image-registry-tls" not found
Oct 30 19:29:49.981391 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.980062    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls podName:153c1fd1-90a2-4063-b066-6503d796d6d8 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:51.980051896 +0000 UTC m=+34.064178304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls") pod "image-registry-5fdc8795f9-cfh7p" (UID: "153c1fd1-90a2-4063-b066-6503d796d6d8") : secret "image-registry-tls" not found
Oct 30 19:29:49.981391 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.980434    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle podName:df3b6a17-4bbf-41dc-83e5-df5172b87b7f nodeName:}" failed. No retries permitted until 2026-10-30 19:29:51.980418867 +0000 UTC m=+34.064545260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle") pod "router-default-6cb4fdcd5-zsxtw" (UID: "df3b6a17-4bbf-41dc-83e5-df5172b87b7f") : configmap references non-existent config key: service-ca.crt
Oct 30 19:29:49.981391 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.980710    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:29:49.981391 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:49.980761    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls podName:f387d9b5-577c-470c-a25e-da28234f464a nodeName:}" failed. No retries permitted until 2026-10-30 19:29:51.980744008 +0000 UTC m=+34.064870415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xdndw" (UID: "f387d9b5-577c-470c-a25e-da28234f464a") : secret "samples-operator-tls" not found
Oct 30 19:29:51.887126 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:51.886130    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hv5j2" event={"ID":"d4c7253d-c52b-4f85-bc06-cfafe6443bb4","Type":"ContainerStarted","Data":"1354efc2b8a404d7c99ae24877989b78b795463786806c920304188642c511f7"}
Oct 30 19:29:51.904303 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:51.904249    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-qjqz6" podStartSLOduration=412.904234897 podStartE2EDuration="6m52.904234897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:49.885129274 +0000 UTC m=+31.969255690" watchObservedRunningTime="2026-10-30 19:29:51.904234897 +0000 UTC m=+33.988361312"
Oct 30 19:29:51.904927 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:51.904888    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hv5j2" podStartSLOduration=365.645604494 podStartE2EDuration="6m7.904877756s" totalImagesPullingTime="2.259273262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:51.903724637 +0000 UTC m=+33.987851056" watchObservedRunningTime="2026-10-30 19:29:51.904877756 +0000 UTC m=+33.989004168"
Oct 30 19:29:52.001414 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.001380    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:29:52.001573 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.001426    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:29:52.001573 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.001457    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:29:52.001573 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.001501    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bwt58\" (UID: \"0c796373-b082-4069-b5e7-228588cc99da\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:29:52.001573 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.001571    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:52.001708 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.001600    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:52.001708 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.001652    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:52.001708 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.001670    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:52.003307 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.001795    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle podName:df3b6a17-4bbf-41dc-83e5-df5172b87b7f nodeName:}" failed. No retries permitted until 2026-10-30 19:29:56.00178243 +0000 UTC m=+38.085908838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle") pod "router-default-6cb4fdcd5-zsxtw" (UID: "df3b6a17-4bbf-41dc-83e5-df5172b87b7f") : configmap references non-existent config key: service-ca.crt
Oct 30 19:29:52.003307 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.002318    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:29:52.003307 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.002391    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls podName:f387d9b5-577c-470c-a25e-da28234f464a nodeName:}" failed. No retries permitted until 2026-10-30 19:29:56.002372769 +0000 UTC m=+38.086499169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xdndw" (UID: "f387d9b5-577c-470c-a25e-da28234f464a") : secret "samples-operator-tls" not found
Oct 30 19:29:52.004128 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.003660    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:29:52.004128 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.003739    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls podName:7427b4f7-fba8-4cde-ad90-1f46d5c92d87 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:56.003718684 +0000 UTC m=+38.087845097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hlvg8" (UID: "7427b4f7-fba8-4cde-ad90-1f46d5c92d87") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:29:52.004128 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.003800    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:29:52.004128 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.003829    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls podName:ac07aa06-d891-4c95-8500-fdef3723ff38 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:56.003819465 +0000 UTC m=+38.087945860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls") pod "dns-default-kl9jq" (UID: "ac07aa06-d891-4c95-8500-fdef3723ff38") : secret "dns-default-metrics-tls" not found
Oct 30 19:29:52.004128 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.003902    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:29:52.004128 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.003912    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7ff684fc5d-nksx9: secret "image-registry-tls" not found
Oct 30 19:29:52.004128 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.003954    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls podName:9e0ec109-980a-4203-b45b-25ebd01ae8ed nodeName:}" failed. No retries permitted until 2026-10-30 19:29:56.003943409 +0000 UTC m=+38.088069820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls") pod "image-registry-7ff684fc5d-nksx9" (UID: "9e0ec109-980a-4203-b45b-25ebd01ae8ed") : secret "image-registry-tls" not found
Oct 30 19:29:52.004128 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.004008    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:29:52.004128 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.004026    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fdc8795f9-cfh7p: secret "image-registry-tls" not found
Oct 30 19:29:52.004128 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.004043    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:29:52.004128 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.004072    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls podName:153c1fd1-90a2-4063-b066-6503d796d6d8 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:56.004059625 +0000 UTC m=+38.088186032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls") pod "image-registry-5fdc8795f9-cfh7p" (UID: "153c1fd1-90a2-4063-b066-6503d796d6d8") : secret "image-registry-tls" not found
Oct 30 19:29:52.004128 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.004112    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert podName:0c796373-b082-4069-b5e7-228588cc99da nodeName:}" failed. No retries permitted until 2026-10-30 19:29:56.004098983 +0000 UTC m=+38.088225392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bwt58" (UID: "0c796373-b082-4069-b5e7-228588cc99da") : secret "networking-console-plugin-cert" not found
Oct 30 19:29:52.004609 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.004159    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:29:52.004609 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.004185    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs podName:df3b6a17-4bbf-41dc-83e5-df5172b87b7f nodeName:}" failed. No retries permitted until 2026-10-30 19:29:56.004177153 +0000 UTC m=+38.088303589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs") pod "router-default-6cb4fdcd5-zsxtw" (UID: "df3b6a17-4bbf-41dc-83e5-df5172b87b7f") : secret "router-metrics-certs-default" not found
Oct 30 19:29:52.102717 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.102615    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:52.109070 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.109033    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:52.204293 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.204251    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkdg\" (UniqueName: \"kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg\") pod \"network-check-target-6wwsm\" (UID: \"926e8fa4-1a22-48b1-832e-ea924ef61178\") " pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:52.204451 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.204387    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs\") pod \"network-metrics-daemon-87b4p\" (UID: \"bb3df8dc-d134-4cee-b95e-21e32f1df218\") " pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:29:52.205108 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.205066    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:29:52.205225 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:52.205152    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs podName:bb3df8dc-d134-4cee-b95e-21e32f1df218 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:24.205129636 +0000 UTC m=+66.289256030 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs") pod "network-metrics-daemon-87b4p" (UID: "bb3df8dc-d134-4cee-b95e-21e32f1df218") : secret "metrics-daemon-secret" not found
Oct 30 19:29:52.214124 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.214046    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzkdg\" (UniqueName: \"kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg\") pod \"network-check-target-6wwsm\" (UID: \"926e8fa4-1a22-48b1-832e-ea924ef61178\") " pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:52.228022 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.227903    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:29:52.246314 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.246002    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:52.546122 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.545761    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4zdqg"]
Oct 30 19:29:52.591227 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.591195    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6wwsm"]
Oct 30 19:29:52.597312 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:52.597206    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926e8fa4_1a22_48b1_832e_ea924ef61178.slice/crio-f14051b645404afad61dd51236779b08cbf4394d7ec80ee2ceae2f731cd3661d WatchSource:0}: Error finding container f14051b645404afad61dd51236779b08cbf4394d7ec80ee2ceae2f731cd3661d: Status 404 returned error can't find the container with id f14051b645404afad61dd51236779b08cbf4394d7ec80ee2ceae2f731cd3661d
Oct 30 19:29:52.891006 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.890928    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6wwsm" event={"ID":"926e8fa4-1a22-48b1-832e-ea924ef61178","Type":"ContainerStarted","Data":"f14051b645404afad61dd51236779b08cbf4394d7ec80ee2ceae2f731cd3661d"}
Oct 30 19:29:52.893133 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:52.893071    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4zdqg" event={"ID":"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e","Type":"ContainerStarted","Data":"dbe267911d17fb48905e78682b379f997883866802948fe4cd4f755ef7d8b749"}
Oct 30 19:29:53.903334 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:53.903209    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4vrds" event={"ID":"6bc162c8-806a-4d1f-a046-3ade90069067","Type":"ContainerStarted","Data":"5769b290662579a73335b2e32a3793bd78ec56674eab6fa8df3897e2724ab13c"}
Oct 30 19:29:53.906427 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:53.905745    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/0.log"
Oct 30 19:29:53.906427 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:53.905786    2569 generic.go:356] "Generic (PLEG): container finished" podID="392e4212-393d-4aaa-904b-f72f1e287c34" containerID="932b774640a6e06e711f949f3866afa063d40df2be7b1b508d9664d473df184c" exitCode=255
Oct 30 19:29:53.906427 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:53.905821    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx" event={"ID":"392e4212-393d-4aaa-904b-f72f1e287c34","Type":"ContainerDied","Data":"932b774640a6e06e711f949f3866afa063d40df2be7b1b508d9664d473df184c"}
Oct 30 19:29:53.906427 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:53.906136    2569 scope.go:122] "RemoveContainer" containerID="932b774640a6e06e711f949f3866afa063d40df2be7b1b508d9664d473df184c"
Oct 30 19:29:53.925158 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:53.925087    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-4vrds" podStartSLOduration=366.348636389 podStartE2EDuration="6m10.92506878s" totalImagesPullingTime="4.576432391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:53.924122025 +0000 UTC m=+36.008248443" watchObservedRunningTime="2026-10-30 19:29:53.92506878 +0000 UTC m=+36.009195199"
Oct 30 19:29:54.341910 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.339947    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8nh5p"]
Oct 30 19:29:54.353495 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.353462    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-pfjv8-8n5cz"]
Oct 30 19:29:54.353673 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.353650    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:54.362675 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.359433    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:29:54.362675 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.359510    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:29:54.362675 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.362641    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zsvdk\""
Oct 30 19:29:54.384911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.383383    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8nh5p"]
Oct 30 19:29:54.384911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.383449    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-pfjv8-8n5cz"]
Oct 30 19:29:54.384911 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.383736    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:29:54.429851 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.429815    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5ff1aae-17b6-4563-949b-e8321be54e25-crio-socket\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:54.430109 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.429905    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:54.430109 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.429994    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:54.430109 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.430049    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5ff1aae-17b6-4563-949b-e8321be54e25-data-volume\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:54.430109 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.430104    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsl6n\" (UniqueName: \"kubernetes.io/projected/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-api-access-jsl6n\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:54.530747 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.530660    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2svq\" (UniqueName: \"kubernetes.io/projected/22a7299c-d539-47fa-87e2-626c5ca8bbbf-kube-api-access-g2svq\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:29:54.530747 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.530709    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:54.530998 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.530794    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5ff1aae-17b6-4563-949b-e8321be54e25-data-volume\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:54.530998 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.530846    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-trusted-ca-bundle\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:29:54.530998 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.530890    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsl6n\" (UniqueName: \"kubernetes.io/projected/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-api-access-jsl6n\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:54.530998 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.530915    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-service-ca-bundle\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:29:54.531172 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.531014    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5ff1aae-17b6-4563-949b-e8321be54e25-crio-socket\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:54.531172 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.531067    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/22a7299c-d539-47fa-87e2-626c5ca8bbbf-archives-path\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:29:54.531172 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.531089    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:54.531542 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.531518    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5ff1aae-17b6-4563-949b-e8321be54e25-data-volume\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:54.531614 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:54.531598    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:29:54.531663 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:54.531657    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls podName:b5ff1aae-17b6-4563-949b-e8321be54e25 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:55.031638969 +0000 UTC m=+37.115765363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8nh5p" (UID: "b5ff1aae-17b6-4563-949b-e8321be54e25") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:29:54.532020 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.531763    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5ff1aae-17b6-4563-949b-e8321be54e25-crio-socket\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:54.532078 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:54.531914    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:29:54.532123 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:54.532112    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-rbac-proxy-cm podName:b5ff1aae-17b6-4563-949b-e8321be54e25 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:55.032095042 +0000 UTC m=+37.116221456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8nh5p" (UID: "b5ff1aae-17b6-4563-949b-e8321be54e25") : configmap "kube-rbac-proxy" not found
Oct 30 19:29:54.556979 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.556826    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsl6n\" (UniqueName: \"kubernetes.io/projected/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-api-access-jsl6n\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:54.635216 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.632061    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/22a7299c-d539-47fa-87e2-626c5ca8bbbf-archives-path\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:29:54.635216 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.632188    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2svq\" (UniqueName: \"kubernetes.io/projected/22a7299c-d539-47fa-87e2-626c5ca8bbbf-kube-api-access-g2svq\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:29:54.635216 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.632288    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-trusted-ca-bundle\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:29:54.635216 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.632325    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-service-ca-bundle\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:29:54.635216 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.633094    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-service-ca-bundle\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:29:54.635216 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.633410    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/22a7299c-d539-47fa-87e2-626c5ca8bbbf-archives-path\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:29:54.635216 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.635106    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-trusted-ca-bundle\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:29:54.666835 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.666803    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2svq\" (UniqueName: \"kubernetes.io/projected/22a7299c-d539-47fa-87e2-626c5ca8bbbf-kube-api-access-g2svq\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:29:54.735276 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.735240    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:29:54.912674 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.912391    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vzpfb" event={"ID":"3615ac3b-837c-46ea-b575-4cdc1edc78a0","Type":"ContainerStarted","Data":"a368402dd484330142f115aa2c2ba3d7bcce3918532429586da157e0efe2ac9a"}
Oct 30 19:29:54.914926 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.914898    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 19:29:54.915401 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.915384    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/0.log"
Oct 30 19:29:54.915495 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.915426    2569 generic.go:356] "Generic (PLEG): container finished" podID="392e4212-393d-4aaa-904b-f72f1e287c34" containerID="7faa03714fb97f461d9afe0d24129275695c5969490602c53524e726085fcc81" exitCode=255
Oct 30 19:29:54.915543 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.915501    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx" event={"ID":"392e4212-393d-4aaa-904b-f72f1e287c34","Type":"ContainerDied","Data":"7faa03714fb97f461d9afe0d24129275695c5969490602c53524e726085fcc81"}
Oct 30 19:29:54.915596 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.915553    2569 scope.go:122] "RemoveContainer" containerID="932b774640a6e06e711f949f3866afa063d40df2be7b1b508d9664d473df184c"
Oct 30 19:29:54.915782 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.915762    2569 scope.go:122] "RemoveContainer" containerID="7faa03714fb97f461d9afe0d24129275695c5969490602c53524e726085fcc81"
Oct 30 19:29:54.916160 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:54.916119    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dp8wx_openshift-console-operator(392e4212-393d-4aaa-904b-f72f1e287c34)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx" podUID="392e4212-393d-4aaa-904b-f72f1e287c34"
Oct 30 19:29:54.921669 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.920984    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv" event={"ID":"468ec5d6-34c5-4577-84a5-ec661a3f2736","Type":"ContainerStarted","Data":"dc7567f9a92d899a1b5c7dda09cd1f370124a0b59768c649797c40c8e8d8bb38"}
Oct 30 19:29:54.924710 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.924648    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk" event={"ID":"4909e20c-2484-41bd-8f06-67f37dfd337b","Type":"ContainerStarted","Data":"c65d83d460dfa1b48b3ab32eabd819f4955e7df61c3c6563239873c559b80285"}
Oct 30 19:29:54.938375 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.937844    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vzpfb" podStartSLOduration=1.289695915 podStartE2EDuration="6.93782494s" totalImagesPullingTime="5.648129025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:54.936667652 +0000 UTC m=+37.020794069" watchObservedRunningTime="2026-10-30 19:29:54.93782494 +0000 UTC m=+37.021951359"
Oct 30 19:29:54.969479 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.966547    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk" podStartSLOduration=357.654913477 podStartE2EDuration="6m2.966527096s" totalImagesPullingTime="5.311613619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:54.959487563 +0000 UTC m=+37.043613980" watchObservedRunningTime="2026-10-30 19:29:54.966527096 +0000 UTC m=+37.050653518"
Oct 30 19:29:54.979325 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:54.979260    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv" podStartSLOduration=360.116555994 podStartE2EDuration="6m4.97924342s" totalImagesPullingTime="4.862687426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:54.978287567 +0000 UTC m=+37.062413983" watchObservedRunningTime="2026-10-30 19:29:54.97924342 +0000 UTC m=+37.063369838"
Oct 30 19:29:55.040617 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.039192    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:55.040617 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.039304    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:55.040617 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:55.040217    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:29:55.040617 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:55.040312    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:29:55.040617 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:55.040361    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-rbac-proxy-cm podName:b5ff1aae-17b6-4563-949b-e8321be54e25 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:56.04025414 +0000 UTC m=+38.124380534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8nh5p" (UID: "b5ff1aae-17b6-4563-949b-e8321be54e25") : configmap "kube-rbac-proxy" not found
Oct 30 19:29:55.040617 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:55.040374    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls podName:b5ff1aae-17b6-4563-949b-e8321be54e25 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:56.040367516 +0000 UTC m=+38.124493910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8nh5p" (UID: "b5ff1aae-17b6-4563-949b-e8321be54e25") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:29:55.068223 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.068188    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-pfjv8-8n5cz"]
Oct 30 19:29:55.070576 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.070549    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7k4fp"]
Oct 30 19:29:55.088348 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:55.088240    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a7299c_d539_47fa_87e2_626c5ca8bbbf.slice/crio-33c7fd5ad315ac4b5b970c81af2a9c37e116079a5627799aa4c7c52b794ed9df WatchSource:0}: Error finding container 33c7fd5ad315ac4b5b970c81af2a9c37e116079a5627799aa4c7c52b794ed9df: Status 404 returned error can't find the container with id 33c7fd5ad315ac4b5b970c81af2a9c37e116079a5627799aa4c7c52b794ed9df
Oct 30 19:29:55.093112 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.093039    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2p5ts"]
Oct 30 19:29:55.094356 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.093203    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7k4fp"
Oct 30 19:29:55.101666 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.101638    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-j4mvd\""
Oct 30 19:29:55.105832 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.104702    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 19:29:55.106182 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.106147    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 19:29:55.107246 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.106767    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7k4fp"]
Oct 30 19:29:55.107246 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.106808    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2p5ts"]
Oct 30 19:29:55.107246 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.106917    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2p5ts"
Oct 30 19:29:55.167277 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.167251    2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4h84r"]
Oct 30 19:29:55.184907 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.184098    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4h84r"]
Oct 30 19:29:55.184907 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.184187    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:29:55.201595 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.201569    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 19:29:55.201825 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.201689    2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xzbzm\""
Oct 30 19:29:55.204348 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.204329    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 19:29:55.204440 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.204348    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 19:29:55.204440 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.204336    2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 19:29:55.242292 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.242209    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99bgq\" (UniqueName: \"kubernetes.io/projected/98fe0d24-2073-47c4-af4a-171c54b7eb2d-kube-api-access-99bgq\") pod \"migrator-66df5d496b-2p5ts\" (UID: \"98fe0d24-2073-47c4-af4a-171c54b7eb2d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2p5ts"
Oct 30 19:29:55.242475 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.242373    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdscq\" (UniqueName: \"kubernetes.io/projected/5db63b46-2839-43a5-9d7d-0cf80e8e1aa7-kube-api-access-fdscq\") pod \"migrator-66df5d496b-7k4fp\" (UID: \"5db63b46-2839-43a5-9d7d-0cf80e8e1aa7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7k4fp"
Oct 30 19:29:55.343296 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.343180    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-signing-key\") pod \"service-ca-5fffc97bdf-4h84r\" (UID: \"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:29:55.343428 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.343330    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99bgq\" (UniqueName: \"kubernetes.io/projected/98fe0d24-2073-47c4-af4a-171c54b7eb2d-kube-api-access-99bgq\") pod \"migrator-66df5d496b-2p5ts\" (UID: \"98fe0d24-2073-47c4-af4a-171c54b7eb2d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2p5ts"
Oct 30 19:29:55.343428 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.343379    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-signing-cabundle\") pod \"service-ca-5fffc97bdf-4h84r\" (UID: \"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:29:55.343488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.343439    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdscq\" (UniqueName: \"kubernetes.io/projected/5db63b46-2839-43a5-9d7d-0cf80e8e1aa7-kube-api-access-fdscq\") pod \"migrator-66df5d496b-7k4fp\" (UID: \"5db63b46-2839-43a5-9d7d-0cf80e8e1aa7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7k4fp"
Oct 30 19:29:55.343488 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.343460    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clprx\" (UniqueName: \"kubernetes.io/projected/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-kube-api-access-clprx\") pod \"service-ca-5fffc97bdf-4h84r\" (UID: \"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:29:55.358008 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.357949    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99bgq\" (UniqueName: \"kubernetes.io/projected/98fe0d24-2073-47c4-af4a-171c54b7eb2d-kube-api-access-99bgq\") pod \"migrator-66df5d496b-2p5ts\" (UID: \"98fe0d24-2073-47c4-af4a-171c54b7eb2d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2p5ts"
Oct 30 19:29:55.360344 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.360316    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdscq\" (UniqueName: \"kubernetes.io/projected/5db63b46-2839-43a5-9d7d-0cf80e8e1aa7-kube-api-access-fdscq\") pod \"migrator-66df5d496b-7k4fp\" (UID: \"5db63b46-2839-43a5-9d7d-0cf80e8e1aa7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7k4fp"
Oct 30 19:29:55.444599 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.444560    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7k4fp"
Oct 30 19:29:55.444841 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.444816    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-signing-cabundle\") pod \"service-ca-5fffc97bdf-4h84r\" (UID: \"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:29:55.445018 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.444931    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clprx\" (UniqueName: \"kubernetes.io/projected/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-kube-api-access-clprx\") pod \"service-ca-5fffc97bdf-4h84r\" (UID: \"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:29:55.445659 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.445137    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-signing-key\") pod \"service-ca-5fffc97bdf-4h84r\" (UID: \"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:29:55.446183 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.446140    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-signing-cabundle\") pod \"service-ca-5fffc97bdf-4h84r\" (UID: \"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:29:55.448519 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.448487    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-signing-key\") pod \"service-ca-5fffc97bdf-4h84r\" (UID: \"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:29:55.457487 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.457456    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clprx\" (UniqueName: \"kubernetes.io/projected/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-kube-api-access-clprx\") pod \"service-ca-5fffc97bdf-4h84r\" (UID: \"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:29:55.474286 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.474066    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2p5ts"
Oct 30 19:29:55.516355 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.516291    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:29:55.767629 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.767546    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7k4fp"]
Oct 30 19:29:55.772380 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:55.772344    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db63b46_2839_43a5_9d7d_0cf80e8e1aa7.slice/crio-281a4c4c3da56a00265e16552431c0542ef51e446b47c275f429272f9e2ba040 WatchSource:0}: Error finding container 281a4c4c3da56a00265e16552431c0542ef51e446b47c275f429272f9e2ba040: Status 404 returned error can't find the container with id 281a4c4c3da56a00265e16552431c0542ef51e446b47c275f429272f9e2ba040
Oct 30 19:29:55.799480 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.799443    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2p5ts"]
Oct 30 19:29:55.804555 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:55.804526    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fe0d24_2073_47c4_af4a_171c54b7eb2d.slice/crio-1b474aa749d00cbc1c774a8548b3527b813e0931b37dbd84958c4db823745aa5 WatchSource:0}: Error finding container 1b474aa749d00cbc1c774a8548b3527b813e0931b37dbd84958c4db823745aa5: Status 404 returned error can't find the container with id 1b474aa749d00cbc1c774a8548b3527b813e0931b37dbd84958c4db823745aa5
Oct 30 19:29:55.873358 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.873323    2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4h84r"]
Oct 30 19:29:55.877532 ip-10-0-138-61 kubenswrapper[2569]: W1030 19:29:55.877501    2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3d3980_29e3_4afa_a9f8_302f62fb0ac4.slice/crio-12639b64583a28bd0527aef31a1e602431f2b40820fb01411be64e6befa78d5e WatchSource:0}: Error finding container 12639b64583a28bd0527aef31a1e602431f2b40820fb01411be64e6befa78d5e: Status 404 returned error can't find the container with id 12639b64583a28bd0527aef31a1e602431f2b40820fb01411be64e6befa78d5e
Oct 30 19:29:55.932598 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.932567    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7k4fp" event={"ID":"5db63b46-2839-43a5-9d7d-0cf80e8e1aa7","Type":"ContainerStarted","Data":"281a4c4c3da56a00265e16552431c0542ef51e446b47c275f429272f9e2ba040"}
Oct 30 19:29:55.935564 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.935292    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pfjv8-8n5cz" event={"ID":"22a7299c-d539-47fa-87e2-626c5ca8bbbf","Type":"ContainerStarted","Data":"037c9e18e762d4c8bfb4e78b80594b52b506a9b16e449a1dfeffa95a699313de"}
Oct 30 19:29:55.935785 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.935561    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pfjv8-8n5cz" event={"ID":"22a7299c-d539-47fa-87e2-626c5ca8bbbf","Type":"ContainerStarted","Data":"33c7fd5ad315ac4b5b970c81af2a9c37e116079a5627799aa4c7c52b794ed9df"}
Oct 30 19:29:55.937232 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.937204    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2p5ts" event={"ID":"98fe0d24-2073-47c4-af4a-171c54b7eb2d","Type":"ContainerStarted","Data":"1b474aa749d00cbc1c774a8548b3527b813e0931b37dbd84958c4db823745aa5"}
Oct 30 19:29:55.943714 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.943689    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 19:29:55.944126 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.944107    2569 scope.go:122] "RemoveContainer" containerID="7faa03714fb97f461d9afe0d24129275695c5969490602c53524e726085fcc81"
Oct 30 19:29:55.944324 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:55.944305    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dp8wx_openshift-console-operator(392e4212-393d-4aaa-904b-f72f1e287c34)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx" podUID="392e4212-393d-4aaa-904b-f72f1e287c34"
Oct 30 19:29:55.946152 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.946118    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j6dm5" event={"ID":"7941072e-d482-42dc-a44e-6e0f9118912f","Type":"ContainerStarted","Data":"e2816b1e7520ffcc86164d30388dc526c1d52c9249574f5446d6ea507acbe21d"}
Oct 30 19:29:55.947768 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:55.947740    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r" event={"ID":"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4","Type":"ContainerStarted","Data":"12639b64583a28bd0527aef31a1e602431f2b40820fb01411be64e6befa78d5e"}
Oct 30 19:29:56.016744 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:56.016421    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-pfjv8-8n5cz" podStartSLOduration=2.016401836 podStartE2EDuration="2.016401836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:56.01447821 +0000 UTC m=+38.098604638" watchObservedRunningTime="2026-10-30 19:29:56.016401836 +0000 UTC m=+38.100528252"
Oct 30 19:29:56.053653 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:56.053615    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:56.053856 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.053771    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:29:56.053856 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:56.053807    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:29:56.053856 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.053834    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs podName:df3b6a17-4bbf-41dc-83e5-df5172b87b7f nodeName:}" failed. No retries permitted until 2026-10-30 19:30:04.053815616 +0000 UTC m=+46.137942032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs") pod "router-default-6cb4fdcd5-zsxtw" (UID: "df3b6a17-4bbf-41dc-83e5-df5172b87b7f") : secret "router-metrics-certs-default" not found
Oct 30 19:29:56.054098 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:56.053882    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:29:56.054098 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.053898    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:29:56.054098 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.053910    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fdc8795f9-cfh7p: secret "image-registry-tls" not found
Oct 30 19:29:56.054098 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.053948    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls podName:153c1fd1-90a2-4063-b066-6503d796d6d8 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:04.053936928 +0000 UTC m=+46.138063322 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls") pod "image-registry-5fdc8795f9-cfh7p" (UID: "153c1fd1-90a2-4063-b066-6503d796d6d8") : secret "image-registry-tls" not found
Oct 30 19:29:56.054098 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:56.053996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:29:56.054098 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:56.054028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:29:56.054098 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:56.054053    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:29:56.054429 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:56.054123    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:56.054429 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:56.054154    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bwt58\" (UID: \"0c796373-b082-4069-b5e7-228588cc99da\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:29:56.054429 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:56.054256    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:29:56.054429 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:56.054316    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:56.054429 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.054419    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:29:56.054680 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.054458    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls podName:b5ff1aae-17b6-4563-949b-e8321be54e25 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:58.054445419 +0000 UTC m=+40.138571827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8nh5p" (UID: "b5ff1aae-17b6-4563-949b-e8321be54e25") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:29:56.054680 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.054521    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:29:56.054680 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.054538    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7ff684fc5d-nksx9: secret "image-registry-tls" not found
Oct 30 19:29:56.054680 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.054584    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls podName:9e0ec109-980a-4203-b45b-25ebd01ae8ed nodeName:}" failed. No retries permitted until 2026-10-30 19:30:04.054568393 +0000 UTC m=+46.138694800 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls") pod "image-registry-7ff684fc5d-nksx9" (UID: "9e0ec109-980a-4203-b45b-25ebd01ae8ed") : secret "image-registry-tls" not found
Oct 30 19:29:56.054680 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.054601    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle podName:df3b6a17-4bbf-41dc-83e5-df5172b87b7f nodeName:}" failed. No retries permitted until 2026-10-30 19:30:04.054592514 +0000 UTC m=+46.138718913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle") pod "router-default-6cb4fdcd5-zsxtw" (UID: "df3b6a17-4bbf-41dc-83e5-df5172b87b7f") : configmap references non-existent config key: service-ca.crt
Oct 30 19:29:56.054956 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.054877    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:29:56.054956 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.054925    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls podName:7427b4f7-fba8-4cde-ad90-1f46d5c92d87 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:04.05491083 +0000 UTC m=+46.139037225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-hlvg8" (UID: "7427b4f7-fba8-4cde-ad90-1f46d5c92d87") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:29:56.055109 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.055090    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:29:56.055159 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.055135    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:29:56.055159 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.055143    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls podName:f387d9b5-577c-470c-a25e-da28234f464a nodeName:}" failed. No retries permitted until 2026-10-30 19:30:04.055127096 +0000 UTC m=+46.139253513 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xdndw" (UID: "f387d9b5-577c-470c-a25e-da28234f464a") : secret "samples-operator-tls" not found
Oct 30 19:29:56.055263 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.055178    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-rbac-proxy-cm podName:b5ff1aae-17b6-4563-949b-e8321be54e25 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:58.055163373 +0000 UTC m=+40.139289768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8nh5p" (UID: "b5ff1aae-17b6-4563-949b-e8321be54e25") : configmap "kube-rbac-proxy" not found
Oct 30 19:29:56.055263 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.055195    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:29:56.055263 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.055225    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls podName:ac07aa06-d891-4c95-8500-fdef3723ff38 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:04.055216177 +0000 UTC m=+46.139342603 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls") pod "dns-default-kl9jq" (UID: "ac07aa06-d891-4c95-8500-fdef3723ff38") : secret "dns-default-metrics-tls" not found
Oct 30 19:29:56.055461 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.055265    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:29:56.055461 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:56.055304    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert podName:0c796373-b082-4069-b5e7-228588cc99da nodeName:}" failed. No retries permitted until 2026-10-30 19:30:04.055292191 +0000 UTC m=+46.139418608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bwt58" (UID: "0c796373-b082-4069-b5e7-228588cc99da") : secret "networking-console-plugin-cert" not found
Oct 30 19:29:56.066142 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:56.066092    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j6dm5" podStartSLOduration=359.957080544 podStartE2EDuration="6m6.066065662s" totalImagesPullingTime="6.108985118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:56.064975152 +0000 UTC m=+38.149101571" watchObservedRunningTime="2026-10-30 19:29:56.066065662 +0000 UTC m=+38.150192079"
Oct 30 19:29:56.965890 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:56.965302    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6wwsm" event={"ID":"926e8fa4-1a22-48b1-832e-ea924ef61178","Type":"ContainerStarted","Data":"c257eb48a8fc426758396efe05a029cb9057b594fd405de49f03dc1f0c494394"}
Oct 30 19:29:56.966798 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:56.966241    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:29:56.970896 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:56.969745    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r" event={"ID":"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4","Type":"ContainerStarted","Data":"59b4d8ef83bec198eacc8870c6e79450878df3a78fe98f990348673b148ad4a9"}
Oct 30 19:29:56.997879 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:56.988670    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6wwsm" podStartSLOduration=34.44685355 podStartE2EDuration="37.98864873s" totalImagesPullingTime="3.54179518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:56.986679243 +0000 UTC m=+39.070805659" watchObservedRunningTime="2026-10-30 19:29:56.98864873 +0000 UTC m=+39.072775147"
Oct 30 19:29:57.010356 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:57.009734    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r" podStartSLOduration=2.009712514 podStartE2EDuration="2.009712514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:57.009528686 +0000 UTC m=+39.093655102" watchObservedRunningTime="2026-10-30 19:29:57.009712514 +0000 UTC m=+39.093838930"
Oct 30 19:29:57.194383 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:57.194282    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2sdcp_b5f201ec-3c4d-4738-b721-ca4e445d63ad/dns-node-resolver/0.log"
Oct 30 19:29:57.974788 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:57.974745    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7k4fp" event={"ID":"5db63b46-2839-43a5-9d7d-0cf80e8e1aa7","Type":"ContainerStarted","Data":"ec8128abb1be36501b7b44c0fdfee047a6c5b815163c2c411357db7d95af1f3a"}
Oct 30 19:29:57.976837 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:57.976805    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2p5ts" event={"ID":"98fe0d24-2073-47c4-af4a-171c54b7eb2d","Type":"ContainerStarted","Data":"62212e5a831ae9304ad2db1d2f677def407cf1d6220be028cfade00eefd62e2c"}
Oct 30 19:29:57.994189 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:57.994143    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7k4fp" podStartSLOduration=1.103135633 podStartE2EDuration="2.994128986s" totalImagesPullingTime="1.890993353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:57.991895273 +0000 UTC m=+40.076021691" watchObservedRunningTime="2026-10-30 19:29:57.994128986 +0000 UTC m=+40.078255401"
Oct 30 19:29:58.010334 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:58.010275    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2p5ts" podStartSLOduration=0.968166503 podStartE2EDuration="3.010255681s" totalImagesPullingTime="2.042089178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:58.009288995 +0000 UTC m=+40.093415407" watchObservedRunningTime="2026-10-30 19:29:58.010255681 +0000 UTC m=+40.094382100"
Oct 30 19:29:58.075954 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:58.075911    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:58.077743 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:58.076065    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:29:58.077743 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:58.076138    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-rbac-proxy-cm podName:b5ff1aae-17b6-4563-949b-e8321be54e25 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:02.076123542 +0000 UTC m=+44.160249948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8nh5p" (UID: "b5ff1aae-17b6-4563-949b-e8321be54e25") : configmap "kube-rbac-proxy" not found
Oct 30 19:29:58.077743 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:58.076176    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:29:58.077743 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:58.076270    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:29:58.077743 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:58.076324    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls podName:b5ff1aae-17b6-4563-949b-e8321be54e25 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:02.076309461 +0000 UTC m=+44.160435876 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8nh5p" (UID: "b5ff1aae-17b6-4563-949b-e8321be54e25") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:29:58.593657 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:58.593626    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lcxk5_da310e19-8c58-4785-a88e-09edc8a63e2e/node-ca/0.log"
Oct 30 19:29:58.648104 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:58.648056    2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:29:58.648289 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:58.648141    2569 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:29:58.648596 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:58.648577    2569 scope.go:122] "RemoveContainer" containerID="7faa03714fb97f461d9afe0d24129275695c5969490602c53524e726085fcc81"
Oct 30 19:29:58.648841 ip-10-0-138-61 kubenswrapper[2569]: E1030 19:29:58.648818    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dp8wx_openshift-console-operator(392e4212-393d-4aaa-904b-f72f1e287c34)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx" podUID="392e4212-393d-4aaa-904b-f72f1e287c34"
Oct 30 19:29:58.993856 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:58.993824    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vzpfb_3615ac3b-837c-46ea-b575-4cdc1edc78a0/serve-healthcheck-canary/0.log"
Oct 30 19:29:59.195647 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:59.195616    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-2p5ts_98fe0d24-2073-47c4-af4a-171c54b7eb2d/migrator/0.log"
Oct 30 19:29:59.394482 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:59.394448    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-7k4fp_5db63b46-2839-43a5-9d7d-0cf80e8e1aa7/migrator/0.log"
Oct 30 19:29:59.595453 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:59.595423    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk_4909e20c-2484-41bd-8f06-67f37dfd337b/kube-storage-version-migrator-operator/0.log"
Oct 30 19:29:59.987532 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:29:59.987441    2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4zdqg" event={"ID":"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e","Type":"ContainerStarted","Data":"3c6438f0ca72aad967575835b458ec66257ff0bf50bc323fa789d6f1f523b47f"}
Oct 30 19:30:00.004253 ip-10-0-138-61 kubenswrapper[2569]: I1030 19:30:00.004203    2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4zdqg" podStartSLOduration=33.846560851 podStartE2EDuration="41.004187469s" totalImagesPullingTime="7.157626618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:00.003315162 +0000 UTC m=+42.087441579" watchObservedRunningTime="2026-10-30 19:30:00.004187469 +0000 UTC m=+42.088313884"
Oct 30 19:30:00.042910 ip-10-0-138-61 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 19:30:00.058629 ip-10-0-138-61 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 19:30:00.058857 ip-10-0-138-61 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 19:30:00.059109 ip-10-0-138-61 systemd[1]: kubelet.service: Consumed 3.444s CPU time.
Oct 30 19:30:00.073662 ip-10-0-138-61 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:30:00.149875 ip-10-0-138-61 kubenswrapper[4823]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:30:00.150449 ip-10-0-138-61 kubenswrapper[4823]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:30:00.150449 ip-10-0-138-61 kubenswrapper[4823]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:30:00.150449 ip-10-0-138-61 kubenswrapper[4823]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:30:00.152149 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152058    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:00.152149 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152148    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:00.152149 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152152    4823 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152156    4823 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152159    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152162    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152165    4823 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152167    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152170    4823 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152173    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152176    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152179    4823 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152182    4823 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152184    4823 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152195    4823 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152198    4823 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152201    4823 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152203    4823 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152206    4823 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152209    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152211    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:00.152268 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152214    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152216    4823 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152219    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152222    4823 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152224    4823 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152227    4823 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152229    4823 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152232    4823 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152235    4823 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152237    4823 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152240    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152243    4823 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152246    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152249    4823 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152252    4823 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152256    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152258    4823 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152261    4823 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152264    4823 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152266    4823 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:00.152757 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152269    4823 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152272    4823 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152274    4823 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152277    4823 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152281    4823 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152284    4823 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152286    4823 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152289    4823 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152291    4823 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152294    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152296    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152299    4823 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152301    4823 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152304    4823 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152307    4823 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152309    4823 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152312    4823 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152314    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152317    4823 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152319    4823 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:00.153323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152322    4823 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152324    4823 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152328    4823 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152331    4823 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152334    4823 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152337    4823 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152339    4823 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152342    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152344    4823 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152347    4823 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152349    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152352    4823 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152356    4823 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152360    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152363    4823 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152366    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152369    4823 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152373    4823 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152375    4823 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:00.153847 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152378    4823 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152381    4823 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152384    4823 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152386    4823 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152389    4823 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152391    4823 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152398    4823 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152401    4823 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152403    4823 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152406    4823 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152409    4823 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152411    4823 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152415    4823 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152418    4823 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152420    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152424    4823 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152426    4823 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152429    4823 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152432    4823 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:00.154369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152437    4823 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152441    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152444    4823 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152447    4823 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152450    4823 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152453    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.152455    4823 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153231    4823 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153240    4823 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153244    4823 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153248    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153251    4823 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153255    4823 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153258    4823 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153261    4823 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153264    4823 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153267    4823 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153269    4823 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153272    4823 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:00.154840 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153277    4823 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153280    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153282    4823 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153287    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153292    4823 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153298    4823 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153303    4823 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153307    4823 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153311    4823 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153315    4823 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153320    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153324    4823 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153329    4823 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153333    4823 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153337    4823 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153342    4823 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153345    4823 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153350    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153354    4823 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:00.155343 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153357    4823 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153362    4823 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153366    4823 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153369    4823 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153373    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153377    4823 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153381    4823 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153386    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153390    4823 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153395    4823 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153398    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153405    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153409    4823 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153415    4823 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153420    4823 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153424    4823 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153429    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153433    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153436    4823 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:00.155812 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153440    4823 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153445    4823 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153449    4823 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153454    4823 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153459    4823 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153463    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153467    4823 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153470    4823 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153472    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153475    4823 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153477    4823 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153480    4823 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153483    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153485    4823 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153488    4823 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153491    4823 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153493    4823 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153496    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153499    4823 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153502    4823 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:00.156301 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153504    4823 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153509    4823 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153512    4823 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153515    4823 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153518    4823 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153520    4823 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153525    4823 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153527    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153530    4823 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153533    4823 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153536    4823 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153538    4823 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153541    4823 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153543    4823 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153546    4823 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153549    4823 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153552    4823 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153554    4823 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153557    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153559    4823 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:00.156831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153562    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153564    4823 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153567    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153569    4823 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153572    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153574    4823 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153577    4823 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153579    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153582    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153584    4823 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153587    4823 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153590    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153592    4823 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153594    4823 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153597    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.153600    4823 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154179    4823 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154187    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154192    4823 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:00.157369 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154195    4823 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154198    4823 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154201    4823 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154204    4823 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154206    4823 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154209    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154212    4823 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154214    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154217    4823 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154220    4823 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154222    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154225    4823 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154227    4823 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154230    4823 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154232    4823 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154235    4823 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154237    4823 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154240    4823 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154242    4823 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:00.157839 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154245    4823 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154248    4823 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154250    4823 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154254    4823 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154259    4823 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154262    4823 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154265    4823 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154267    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154271    4823 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154273    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154276    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154279    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154281    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154284    4823 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154287    4823 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154289    4823 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154292    4823 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154294    4823 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154297    4823 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:00.158513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154300    4823 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154304    4823 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154308    4823 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154311    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154313    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154317    4823 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154320    4823 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154322    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154325    4823 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154328    4823 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154330    4823 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154333    4823 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154336    4823 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154339    4823 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154342    4823 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154344    4823 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154347    4823 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154350    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154352    4823 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154355    4823 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:00.159021 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154357    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154360    4823 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154363    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154366    4823 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154368    4823 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154371    4823 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154374    4823 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154377    4823 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154379    4823 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154382    4823 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154386    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154388    4823 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154391    4823 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154394    4823 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154397    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154399    4823 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154402    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154404    4823 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154407    4823 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154410    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:00.159539 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154413    4823 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154415    4823 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154418    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154420    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154423    4823 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154425    4823 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154428    4823 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154430    4823 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154433    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154435    4823 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154438    4823 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154440    4823 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154442    4823 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154445    4823 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154447    4823 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154451    4823 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154456    4823 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154458    4823 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154461    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154464    4823 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:00.160247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154466    4823 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154470    4823 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154472    4823 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154475    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.154477    4823 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154592    4823 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154600    4823 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154609    4823 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154616    4823 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154624    4823 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154629    4823 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154634    4823 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154638    4823 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154641    4823 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154645    4823 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154648    4823 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154652    4823 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154655    4823 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154658    4823 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154661    4823 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154664    4823 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154667    4823 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154670    4823 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:30:00.161169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154675    4823 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154678    4823 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154681    4823 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154685    4823 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154688    4823 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154695    4823 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154699    4823 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154702    4823 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154705    4823 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154709    4823 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154712    4823 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154715    4823 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154718    4823 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154721    4823 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154726    4823 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154729    4823 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154732    4823 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154735    4823 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154738    4823 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154741    4823 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154746    4823 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154749    4823 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154752    4823 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154755    4823 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:30:00.162170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154758    4823 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154763    4823 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154766    4823 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154769    4823 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154773    4823 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154776    4823 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154779    4823 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154782    4823 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154785    4823 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154789    4823 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154792    4823 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154795    4823 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154800    4823 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154805    4823 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154808    4823 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154812    4823 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154815    4823 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154818    4823 flags.go:64] FLAG: --help="false"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154821    4823 flags.go:64] FLAG: --hostname-override="ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154824    4823 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154829    4823 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154832    4823 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154835    4823 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:30:00.163183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154839    4823 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154842    4823 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154846    4823 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154849    4823 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154852    4823 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154855    4823 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154871    4823 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154874    4823 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154878    4823 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154881    4823 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154885    4823 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154888    4823 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154891    4823 flags.go:64] FLAG: --lock-file=""
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154894    4823 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154897    4823 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154901    4823 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154907    4823 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154910    4823 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154913    4823 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154916    4823 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154919    4823 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154922    4823 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154929    4823 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154932    4823 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154937    4823 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:30:00.164226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154941    4823 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154945    4823 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154948    4823 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154951    4823 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154956    4823 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154959    4823 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154962    4823 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154965    4823 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154968    4823 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154976    4823 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154979    4823 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154982    4823 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154985    4823 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154988    4823 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154991    4823 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154994    4823 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.154998    4823 flags.go:64] FLAG: --port="10250"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155001    4823 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155004    4823 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-088203cde8f433334"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155007    4823 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155011    4823 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155014    4823 flags.go:64] FLAG: --register-node="true"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155017    4823 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155021    4823 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155024    4823 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155026    4823 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:30:00.165327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155029    4823 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155033    4823 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155037    4823 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155042    4823 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155045    4823 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155048    4823 flags.go:64] FLAG: --runonce="false"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155052    4823 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155055    4823 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155058    4823 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155061    4823 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155066    4823 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155069    4823 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155072    4823 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155075    4823 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155078    4823 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155081    4823 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155084    4823 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155087    4823 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155090    4823 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155093    4823 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155096    4823 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155101    4823 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155104    4823 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155107    4823 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155112    4823 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:30:00.166510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155115    4823 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155118    4823 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155121    4823 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155124    4823 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155127    4823 flags.go:64] FLAG: --v="2"
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155131    4823 flags.go:64] FLAG: --version="false"
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155135    4823 flags.go:64] FLAG: --vmodule=""
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155140    4823 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155143    4823 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155303    4823 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155309    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155312    4823 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155315    4823 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155318    4823 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155322    4823 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155324    4823 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155327    4823 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155332    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155334    4823 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155337    4823 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155340    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155342    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:00.167558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155345    4823 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155349    4823 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155351    4823 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155354    4823 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155357    4823 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155360    4823 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155363    4823 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155365    4823 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155368    4823 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155371    4823 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155373    4823 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155375    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155378    4823 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155381    4823 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155384    4823 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155387    4823 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155389    4823 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155392    4823 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155394    4823 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155397    4823 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:00.168477 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155401    4823 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155403    4823 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155406    4823 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155409    4823 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155412    4823 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155421    4823 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155425    4823 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155429    4823 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155432    4823 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155438    4823 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155440    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155443    4823 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155445    4823 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155448    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155451    4823 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155453    4823 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155456    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155458    4823 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155461    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:00.169120 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155464    4823 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155466    4823 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155469    4823 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155471    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155474    4823 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155477    4823 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155480    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155482    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155485    4823 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155487    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155490    4823 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155492    4823 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155495    4823 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155499    4823 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155501    4823 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155504    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155507    4823 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155509    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155512    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155521    4823 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:00.169593 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155525    4823 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155528    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155530    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155533    4823 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155536    4823 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155538    4823 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155541    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155544    4823 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155547    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155549    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155552    4823 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155555    4823 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155557    4823 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155560    4823 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155562    4823 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155565    4823 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155567    4823 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155570    4823 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155573    4823 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:00.170108 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155575    4823 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:00.170572 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155578    4823 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:00.170572 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155580    4823 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:00.170572 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155584    4823 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:00.170572 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155588    4823 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:00.170572 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155591    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:00.170572 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155595    4823 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:00.170572 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155598    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:00.170572 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155601    4823 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:00.170572 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155604    4823 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:00.170572 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155607    4823 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:00.170572 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155610    4823 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:00.170572 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155612    4823 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:00.170572 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155622    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:00.170572 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155625    4823 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:00.170572 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.155653    4823 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:30:00.170572 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155696    4823 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155700    4823 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155703    4823 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155706    4823 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155709    4823 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155717    4823 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155720    4823 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155723    4823 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155725    4823 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155729    4823 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155734    4823 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155737    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155740    4823 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155743    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155745    4823 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155749    4823 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155751    4823 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155754    4823 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155757    4823 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:00.170990 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155759    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155762    4823 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155765    4823 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155767    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155772    4823 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155774    4823 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155777    4823 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155779    4823 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155782    4823 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155785    4823 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155789    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155797    4823 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155800    4823 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155803    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155805    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155808    4823 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155811    4823 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155814    4823 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155816    4823 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155819    4823 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:00.171453 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155822    4823 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155825    4823 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155828    4823 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155830    4823 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155833    4823 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155836    4823 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155838    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155841    4823 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155843    4823 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155846    4823 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155848    4823 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155851    4823 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155853    4823 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155856    4823 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155871    4823 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155874    4823 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155879    4823 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155881    4823 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155884    4823 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:00.171969 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155886    4823 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155890    4823 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155893    4823 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155897    4823 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155900    4823 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155906    4823 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155909    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155912    4823 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155914    4823 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155917    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155919    4823 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155922    4823 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155925    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155927    4823 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155930    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155932    4823 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155935    4823 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155938    4823 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155940    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:00.172430 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155943    4823 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155945    4823 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155949    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155952    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155954    4823 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155957    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155959    4823 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155962    4823 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155965    4823 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155967    4823 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155972    4823 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155975    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155978    4823 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155980    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155983    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155986    4823 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155990    4823 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155992    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155995    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:00.173059 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.155998    4823 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.156001    4823 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.156003    4823 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.156006    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.156009    4823 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.156012    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.156014    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.156017    4823 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.156020    4823 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.156022    4823 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.162518    4823 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.162752    4823 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.162778    4823 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.162787    4823 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162903    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162913    4823 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162918    4823 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162924    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162929    4823 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162933    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:00.173517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162938    4823 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162942    4823 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162946    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162952    4823 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162956    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162961    4823 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162965    4823 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162968    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162973    4823 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162977    4823 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162982    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162986    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162991    4823 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.162995    4823 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163000    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163004    4823 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163009    4823 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163013    4823 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163018    4823 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163022    4823 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:00.174028 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163026    4823 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163031    4823 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163038    4823 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163048    4823 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163052    4823 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163057    4823 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163061    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163066    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163070    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163074    4823 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163079    4823 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163083    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163088    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163092    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163096    4823 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163101    4823 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163105    4823 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163110    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163114    4823 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163118    4823 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:00.174521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163122    4823 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163127    4823 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163131    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163135    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163140    4823 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163144    4823 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163149    4823 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163153    4823 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163157    4823 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163162    4823 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163166    4823 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163171    4823 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163176    4823 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163181    4823 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163185    4823 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163189    4823 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163194    4823 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163198    4823 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163202    4823 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:00.175037 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163206    4823 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163210    4823 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163216    4823 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163223    4823 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163228    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163233    4823 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163238    4823 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163242    4823 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163247    4823 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163252    4823 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163256    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163261    4823 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163265    4823 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163270    4823 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163275    4823 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163279    4823 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163284    4823 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163289    4823 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163293    4823 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163297    4823 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:00.175497 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163302    4823 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163306    4823 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163311    4823 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163315    4823 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163320    4823 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163324    4823 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163328    4823 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163332    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163337    4823 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163341    4823 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163347    4823 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163351    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163356    4823 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163360    4823 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163364    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163368    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163373    4823 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163377    4823 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163381    4823 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:00.176088 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163385    4823 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:00.176558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163390    4823 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:00.176558 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.163441    4823 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:30:00.176558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163677    4823 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:00.176558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163685    4823 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:00.176558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163691    4823 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:00.176558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163696    4823 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:00.176558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163700    4823 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:00.176558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163704    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:00.176558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163708    4823 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:00.176558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163713    4823 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:00.176558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163717    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:00.176558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163722    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:00.176558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163728    4823 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:00.176558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163732    4823 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:00.176558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163736    4823 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:00.176558 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163742    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163746    4823 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163751    4823 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163755    4823 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163760    4823 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163764    4823 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163769    4823 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163773    4823 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163779    4823 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163783    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163787    4823 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163792    4823 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163796    4823 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163801    4823 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163805    4823 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163810    4823 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163815    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163819    4823 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163823    4823 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163827    4823 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:00.177155 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163832    4823 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163836    4823 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163840    4823 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163844    4823 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163849    4823 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163853    4823 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163873    4823 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163878    4823 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163882    4823 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163887    4823 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163892    4823 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163896    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163900    4823 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163904    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163910    4823 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163914    4823 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163918    4823 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163923    4823 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163927    4823 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163932    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:00.177734 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163936    4823 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163942    4823 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163946    4823 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163951    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163955    4823 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163959    4823 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163963    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163968    4823 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163972    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163976    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163981    4823 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163986    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.163990    4823 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164028    4823 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164033    4823 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164037    4823 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164041    4823 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164046    4823 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164051    4823 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164056    4823 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:00.178277 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164061    4823 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164066    4823 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164070    4823 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164074    4823 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164079    4823 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164084    4823 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164090    4823 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164096    4823 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164100    4823 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164105    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164109    4823 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164114    4823 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164118    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164123    4823 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164129    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164134    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164139    4823 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164143    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164148    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:00.178822 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164153    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:00.179318 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164157    4823 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:00.179318 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164164    4823 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:00.179318 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164170    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:00.179318 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164175    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:00.179318 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164180    4823 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:00.179318 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164185    4823 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:00.179318 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164189    4823 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:00.179318 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164193    4823 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:00.179318 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164198    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:00.179318 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164202    4823 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:00.179318 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164207    4823 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:00.179318 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164212    4823 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:00.179318 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164216    4823 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:00.179318 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.164264    4823 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:30:00.179318 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164319    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164327    4823 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164332    4823 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164336    4823 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164341    4823 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164346    4823 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164351    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164355    4823 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164359    4823 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164364    4823 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164368    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164373    4823 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164377    4823 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164382    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164386    4823 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164390    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164395    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164400    4823 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164404    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:30:00.179712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164408    4823 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164412    4823 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164417    4823 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164421    4823 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164426    4823 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164430    4823 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164435    4823 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164466    4823 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164472    4823 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164476    4823 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164481    4823 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164485    4823 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164490    4823 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164495    4823 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164501    4823 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164507    4823 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164512    4823 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164517    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164522    4823 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164526    4823 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:30:00.180281 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164531    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164535    4823 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164540    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164544    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164549    4823 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164553    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164558    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164563    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164568    4823 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164576    4823 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164584    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164589    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164594    4823 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164598    4823 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164603    4823 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164607    4823 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164611    4823 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164616    4823 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164620    4823 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:30:00.180816 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164624    4823 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164629    4823 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164633    4823 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164637    4823 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164641    4823 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164646    4823 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164650    4823 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164654    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164658    4823 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164663    4823 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164667    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164671    4823 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164676    4823 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164680    4823 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164685    4823 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164689    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164694    4823 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164698    4823 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164702    4823 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:30:00.181316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164707    4823 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164712    4823 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164717    4823 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164721    4823 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164725    4823 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164729    4823 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164737    4823 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164741    4823 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164746    4823 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164751    4823 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164755    4823 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164761    4823 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164767    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164772    4823 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164776    4823 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164781    4823 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164786    4823 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164790    4823 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164794    4823 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164799    4823 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:30:00.181792 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164804    4823 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:30:00.182336 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164808    4823 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:30:00.182336 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164812    4823 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:30:00.182336 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164816    4823 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:30:00.182336 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164821    4823 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:30:00.182336 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164825    4823 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:30:00.182336 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164829    4823 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:30:00.182336 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164833    4823 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:30:00.182336 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:00.164837    4823 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:30:00.182336 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.164947    4823 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:30:00.182336 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.166602    4823 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 19:30:00.182336 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.166680    4823 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 19:30:00.182336 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.166993    4823 server.go:1014] "Starting client certificate rotation"
Oct 30 19:30:00.182336 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.167114    4823 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:30:00.182336 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.167510    4823 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:24:18 +0000 UTC" deadline="2028-07-11 11:59:38.07226235 +0000 UTC"
Oct 30 19:30:00.182336 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.168189    4823 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14872h29m37.904079107s"
Oct 30 19:30:00.182336 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.169243    4823 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:30:00.203485 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.203451    4823 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:30:00.205171 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.205153    4823 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:30:00.206921 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.206905    4823 log.go:25] "Validated CRI v1 image API"
Oct 30 19:30:00.207699 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.207683    4823 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:30:00.209764 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.209737    4823 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8d1b0409-7b13-43bc-a7f3-6bc62e421630:/dev/nvme0n1p4 cd23f126-2bbd-49ef-98c2-6d8fd187918c:/dev/nvme0n1p3]
Oct 30 19:30:00.210042 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.209765    4823 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0166d3b7fd00652d2525020761ad62349e7fcd17cb6c5866bd5633d9827af5f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0166d3b7fd00652d2525020761ad62349e7fcd17cb6c5866bd5633d9827af5f5/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b859724c833e248f69a3b21c3a0ed8a9270ce2a8e686d446d0856ad8f0392a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b859724c833e248f69a3b21c3a0ed8a9270ce2a8e686d446d0856ad8f0392a2/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12639b64583a28bd0527aef31a1e602431f2b40820fb01411be64e6befa78d5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12639b64583a28bd0527aef31a1e602431f2b40820fb01411be64e6befa78d5e/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12b4dd3d73c6a1df2d18d4e074867d79373af5546eafec150feaf4aa45bad2ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12b4dd3d73c6a1df2d18d4e074867d79373af5546eafec150feaf4aa45bad2ab/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14ac690ea497ca589a3c1266822bb4e2e626396b5b84648b4d25e101985876f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14ac690ea497ca589a3c1266822bb4e2e626396b5b84648b4d25e101985876f9/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/159873945b00a7e58f88326c67234600440fdacd52e6967ad11414cc0039e831/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/159873945b00a7e58f88326c67234600440fdacd52e6967ad11414cc0039e831/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17bebd71442cee57df0a63815d5d2ae487a270b957f1bfe226ce7d9475b88fe3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17bebd71442cee57df0a63815d5d2ae487a270b957f1bfe226ce7d9475b88fe3/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b474aa749d00cbc1c774a8548b3527b813e0931b37dbd84958c4db823745aa5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b474aa749d00cbc1c774a8548b3527b813e0931b37dbd84958c4db823745aa5/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/234270c9f293615bc63d23ff406cb11dc35111515be70d1f31c5a069a2e9f8be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/234270c9f293615bc63d23ff406cb11dc35111515be70d1f31c5a069a2e9f8be/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24eb387a935fbe718617009ea1258f85c1430143dda2b1b9ac47b8f4e16d4e35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24eb387a935fbe718617009ea1258f85c1430143dda2b1b9ac47b8f4e16d4e35/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/281a4c4c3da56a00265e16552431c0542ef51e446b47c275f429272f9e2ba040/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/281a4c4c3da56a00265e16552431c0542ef51e446b47c275f429272f9e2ba040/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/339613c0d5b9ef3b19a2724086dfcd73a391e38e746e7a49802e28c44478c19d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/339613c0d5b9ef3b19a2724086dfcd73a391e38e746e7a49802e28c44478c19d/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33c7fd5ad315ac4b5b970c81af2a9c37e116079a5627799aa4c7c52b794ed9df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33c7fd5ad315ac4b5b970c81af2a9c37e116079a5627799aa4c7c52b794ed9df/userdata/shm major:0 minor:329 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/343d60461be4bd745b9eb08058f432ffb36a1cc7bf0f4822efb0101a0359b6de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/343d60461be4bd745b9eb08058f432ffb36a1cc7bf0f4822efb0101a0359b6de/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f1402c6a45b330bbb1cc8cd1e011283fcbb6dbc621e3c8679a903e3fa69bbd0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f1402c6a45b330bbb1cc8cd1e011283fcbb6dbc621e3c8679a903e3fa69bbd0/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7cdfd04130d7c43badbb4acabd61789fcc62d63f0b04bade27b9fcd4876c1cdb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7cdfd04130d7c43badbb4acabd61789fcc62d63f0b04bade27b9fcd4876c1cdb/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89878ff87d7d6936af6e057d953f2858ab03f66e9789b28558a113c376fe20d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89878ff87d7d6936af6e057d953f2858ab03f66e9789b28558a113c376fe20d5/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a9e7044a68634c91220db20574d26df0b9c05fb61bed3060d0df1effb170ace/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a9e7044a68634c91220db20574d26df0b9c05fb61bed3060d0df1effb170ace/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7a16614b766a578de04bc5f01c9de025ad846dc68802f7bae453eef46f936e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7a16614b766a578de04bc5f01c9de025ad846dc68802f7bae453eef46f936e8/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf58f46fc376b90dd9b3c55c60be823fca4491f1fe37b32845f586a907dbac35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf58f46fc376b90dd9b3c55c60be823fca4491f1fe37b32845f586a907dbac35/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8b94627ee953e7c239fc66fbf71993f6c710c9eacdb2763ca70ef6aaefc809f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8b94627ee953e7c239fc66fbf71993f6c710c9eacdb2763ca70ef6aaefc809f/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbe267911d17fb48905e78682b379f997883866802948fe4cd4f755ef7d8b749/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbe267911d17fb48905e78682b379f997883866802948fe4cd4f755ef7d8b749/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de4b10692573ed031d00c1d6fcb29945422022be92d25e9d2e6f7c4be83948b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de4b10692573ed031d00c1d6fcb29945422022be92d25e9d2e6f7c4be83948b9/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7bdcfe753eeaad93d91080684857987cc8d9e8813f718e927802a0681d5902b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7bdcfe753eeaad93d91080684857987cc8d9e8813f718e927802a0681d5902b/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f14051b645404afad61dd51236779b08cbf4394d7ec80ee2ceae2f731cd3661d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f14051b645404afad61dd51236779b08cbf4394d7ec80ee2ceae2f731cd3661d/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f87225c4e0d6745d6509a153c22157103d8b00e9661b92b940d95e3fa9abbf23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f87225c4e0d6745d6509a153c22157103d8b00e9661b92b940d95e3fa9abbf23/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0af906a5-4f1e-4feb-8e1d-366cce9964c0/volumes/kubernetes.io~projected/kube-api-access-rzjfz:{mountpoint:/var/lib/kubelet/pods/0af906a5-4f1e-4feb-8e1d-366cce9964c0/volumes/kubernetes.io~projected/kube-api-access-rzjfz major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/153c1fd1-90a2-4063-b066-6503d796d6d8/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/153c1fd1-90a2-4063-b066-6503d796d6d8/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/153c1fd1-90a2-4063-b066-6503d796d6d8/volumes/kubernetes.io~projected/kube-api-access-zfdmm:{mountpoint:/var/lib/kubelet/pods/153c1fd1-90a2-4063-b066-6503d796d6d8/volumes/kubernetes.io~projected/kube-api-access-zfdmm major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/153c1fd1-90a2-4063-b066-6503d796d6d8/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/153c1fd1-90a2-4063-b066-6503d796d6d8/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/153c1fd1-90a2-4063-b066-6503d796d6d8/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/153c1fd1-90a2-4063-b066-6503d796d6d8/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22a7299c-d539-47fa-87e2-626c5ca8bbbf/volumes/kubernetes.io~projected/kube-api-access-g2svq:{mountpoint:/var/lib/kubelet/pods/22a7299c-d539-47fa-87e2-626c5ca8bbbf/volumes/kubernetes.io~projected/kube-api-access-g2svq major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28217977-99fe-4540-a322-3b3efb6dfa36/volumes/kubernetes.io~projected/kube-api-access-2bbgk:{mountpoint:/var/lib/kubelet/pods/28217977-99fe-4540-a322-3b3efb6dfa36/volumes/kubernetes.io~projected/kube-api-access-2bbgk major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3615ac3b-837c-46ea-b575-4cdc1edc78a0/volumes/kubernetes.io~projected/kube-api-access-jhbv6:{mountpoint:/var/lib/kubelet/pods/3615ac3b-837c-46ea-b575-4cdc1edc78a0/volumes/kubernetes.io~projected/kube-api-access-jhbv6 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3615ac3b-837c-46ea-b575-4cdc1edc78a0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3615ac3b-837c-46ea-b575-4cdc1edc78a0/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39126eba-223c-47c2-aff5-4233035260a3/volumes/kubernetes.io~projected/kube-api-access-7xd7c:{mountpoint:/var/lib/kubelet/pods/39126eba-223c-47c2-aff5-4233035260a3/volumes/kubernetes.io~projected/kube-api-access-7xd7c major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/392e4212-393d-4aaa-904b-f72f1e287c34/volumes/kubernetes.io~projected/kube-api-access-kn4pl:{mountpoint:/var/lib/kubelet/pods/392e4212-393d-4aaa-904b-f72f1e287c34/volumes/kubernetes.io~projected/kube-api-access-kn4pl major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/392e4212-393d-4aaa-904b-f72f1e287c34/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/392e4212-393d-4aaa-904b-f72f1e287c34/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/468ec5d6-34c5-4577-84a5-ec661a3f2736/volumes/kubernetes.io~projected/kube-api-access-h6qvt:{mountpoint:/var/lib/kubelet/pods/468ec5d6-34c5-4577-84a5-ec661a3f2736/volumes/kubernetes.io~projected/kube-api-access-h6qvt major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/468ec5d6-34c5-4577-84a5-ec661a3f2736/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/468ec5d6-34c5-4577-84a5-ec661a3f2736/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4909e20c-2484-41bd-8f06-67f37dfd337b/volumes/kubernetes.io~projected/kube-api-access-xjmx9:{mountpoint:/var/lib/kubelet/pods/4909e20c-2484-41bd-8f06-67f37dfd337b/volumes/kubernetes.io~projected/kube-api-access-xjmx9 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4909e20c-2484-41bd-8f06-67f37dfd337b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4909e20c-2484-41bd-8f06-67f37dfd337b/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52efc11e-b616-4c33-97d8-06fed7d75f82/volumes/kubernetes.io~projected/kube-api-access-pcfz2:{mountpoint:/var/lib/kubelet/pods/52efc11e-b616-4c33-97d8-06fed7d75f82/volumes/kubernetes.io~projected/kube-api-access-pcfz2 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5db63b46-2839-43a5-9d7d-0cf80e8e1aa7/volumes/kubernetes.io~projected/kube-api-access-fdscq:{mountpoint:/var/lib/kubelet/pods/5db63b46-2839-43a5-9d7d-0cf80e8e1aa7/volumes/kubernetes.io~projected/kube-api-access-fdscq major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bc162c8-806a-4d1f-a046-3ade90069067/volumes/kubernetes.io~projected/kube-api-access-mp8kb:{mountpoint:/var/lib/kubelet/pods/6bc162c8-806a-4d1f-a046-3ade90069067/volumes/kubernetes.io~projected/kube-api-access-mp8kb major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bc162c8-806a-4d1f-a046-3ade90069067/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6bc162c8-806a-4d1f-a046-3ade90069067/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c/volumes/kubernetes.io~projected/kube-api-access-cs72w:{mountpoint:/var/lib/kubelet/pods/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c/volumes/kubernetes.io~projected/kube-api-access-cs72w major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7427b4f7-fba8-4cde-ad90-1f46d5c92d87/volumes/kubernetes.io~projected/kube-api-access-cb4cf:{mountpoint:/var/lib/kubelet/pods/7427b4f7-fba8-4cde-ad90-1f46d5c92d87/volumes/kubernetes.io~projected/kube-api-access-cb4cf major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7941072e-d482-42dc-a44e-6e0f9118912f/volumes/kubernetes.io~projected/kube-api-access-884tr:{mountpoint:/var/lib/kubelet/pods/7941072e-d482-42dc-a44e-6e0f9118912f/volumes/kubernetes.io~projected/kube-api-access-884tr major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/926e8fa4-1a22-48b1-832e-ea924ef61178/volumes/kubernetes.io~projected/kube-api-access-rzkdg:{mountpoint:/var/lib/kubelet/pods/926e8fa4-1a22-48b1-832e-ea924ef61178/volumes/kubernetes.io~projected/kube-api-access-rzkdg major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98fe0d24-2073-47c4-af4a-171c54b7eb2d/volumes/kubernetes.io~projected/kube-api-access-99bgq:{mountpoint:/var/lib/kubelet/pods/98fe0d24-2073-47c4-af4a-171c54b7eb2d/volumes/kubernetes.io~projected/kube-api-access-99bgq major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e0ec109-980a-4203-b45b-25ebd01ae8ed/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9e0ec109-980a-4203-b45b-25ebd01ae8ed/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e0ec109-980a-4203-b45b-25ebd01ae8ed/volumes/kubernetes.io~projected/kube-api-access-6b8qf:{mountpoint:/var/lib/kubelet/pods/9e0ec109-980a-4203-b45b-25ebd01ae8ed/volumes/kubernetes.io~projected/kube-api-access-6b8qf major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e0ec109-980a-4203-b45b-25ebd01ae8ed/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9e0ec109-980a-4203-b45b-25ebd01ae8ed/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e0ec109-980a-4203-b45b-25ebd01ae8ed/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9e0ec109-980a-4203-b45b-25ebd01ae8ed/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac07aa06-d891-4c95-8500-fdef3723ff38/volumes/kubernetes.io~projected/kube-api-access-5vxfk:{mountpoint:/var/lib/kubelet/pods/ac07aa06-d891-4c95-8500-fdef3723ff38/volumes/kubernetes.io~projected/kube-api-access-5vxfk major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5f201ec-3c4d-4738-b721-ca4e445d63ad/volumes/kubernetes.io~projected/kube-api-access-l6hxs:{mountpoint:/var/lib/kubelet/pods/b5f201ec-3c4d-4738-b721-ca4e445d63ad/volumes/kubernetes.io~projected/kube-api-access-l6hxs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5ff1aae-17b6-4563-949b-e8321be54e25/volumes/kubernetes.io~projected/kube-api-access-jsl6n:{mountpoint:/var/lib/kubelet/pods/b5ff1aae-17b6-4563-949b-e8321be54e25/volumes/kubernetes.io~projected/kube-api-access-jsl6n major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb3df8dc-d134-4cee-b95e-21e32f1df218/volumes/kubernetes.io~projected/kube-api-access-8zgxv:{mountpoint:/var/lib/kubelet/pods/bb3df8dc-d134-4cee-b95e-21e32f1df218/volumes/kubernetes.io~projected/kube-api-access-8zgxv major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c246f08a-6fa3-4664-8a73-567d35f22d3a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c246f08a-6fa3-4664-8a73-567d35f22d3a/volumes/kubernetes.io~secret/agent-certs major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6ec634f-81f4-4d4e-97ef-33092b38052a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c6ec634f-81f4-4d4e-97ef-33092b38052a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6ec634f-81f4-4d4e-97ef-33092b38052a/volumes/kubernetes.io~projected/kube-api-access-22dvx:{mountpoint:/var/lib/kubelet/pods/c6ec634f-81f4-4d4e-97ef-33092b38052a/volumes/kubernetes.io~projected/kube-api-access-22dvx major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6ec634f-81f4-4d4e-97ef-33092b38052a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c6ec634f-81f4-4d4e-97ef-33092b38052a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4/volumes/kubernetes.io~projected/kube-api-access-clprx:{mountpoint:/var/lib/kubelet/pods/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4/volumes/kubernetes.io~projected/kube-api-access-clprx major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4/volumes/kubernetes.io~secret/signing-key major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4c7253d-c52b-4f85-bc06-cfafe6443bb4/volumes/kubernetes.io~projected/kube-api-access-gtz66:{mountpoint:/var/lib/kubelet/pods/d4c7253d-c52b-4f85-bc06-cfafe6443bb4/volumes/kubernetes.io~projected/kube-api-access-gtz66 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5/volumes/kubernetes.io~projected/kube-api-access-gkz6g:{mountpoint:/var/lib/kubelet/pods/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5/volumes/kubernetes.io~projected/kube-api-access-gkz6g major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da310e19-8c58-4785-a88e-09edc8a63e2e/volumes/kubernetes.io~projected/kube-api-access-4wkt8:{mountpoint:/var/lib/kubelet/pods/da310e19-8c58-4785-a88e-09edc8a63e2e/volumes/kubernetes.io~projected/kube-api-access-4wkt8 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df3b6a17-4bbf-41dc-83e5-df5172b87b7f/volumes/kubernetes.io~projected/kube-api-access-5q76f:{mountpoint:/var/lib/kubelet/pods/df3b6a17-4bbf-41dc-83e5-df5172b87b7f/volumes/kubernetes.io~projected/kube-api-access-5q76f major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df3b6a17-4bbf-41dc-83e5-df5172b87b7f/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/df3b6a17-4bbf-41dc-83e5-df5172b87b7f/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df3b6a17-4bbf-41dc-83e5-df5172b87b7f/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/df3b6a17-4bbf-41dc-83e5-df5172b87b7f/volumes/kubernetes.io~secret/stats-auth major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e683558b-f046-46ae-99c4-4862e1247b03/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e683558b-f046-46ae-99c4-4862e1247b03/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e683558b-f046-46ae-99c4-4862e1247b03/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e683558b-f046-46ae-99c4-4862e1247b03/volumes/kubernetes.io~empty-dir/tmp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e683558b-f046-46ae-99c4-4862e1247b03/volumes/kubernetes.io~projected/kube-api-access-729fn:{mountpoint:/var/lib/kubelet/pods/e683558b-f046-46ae-99c4-4862e1247b03/volumes/kubernetes.io~projected/kube-api-access-729fn major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebaeaca4-4212-4bcc-b979-873495265bad/volumes/kubernetes.io~projected/kube-api-access-72lkw:{mountpoint:/var/lib/kubelet/pods/ebaeaca4-4212-4bcc-b979-873495265bad/volumes/kubernetes.io~projected/kube-api-access-72lkw major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f387d9b5-577c-470c-a25e-da28234f464a/volumes/kubernetes.io~projected/kube-api-access-8s4j8:{mountpoint:/var/lib/kubelet/pods/f387d9b5-577c-470c-a25e-da28234f464a/volumes/kubernetes.io~projected/kube-api-access-8s4j8 major:0 minor:214 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/75f84e65e9d104323918098063aa6a1eb02f30ed5ea122798298e3c658ad1cb5/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/7976e7508dc99ca31e77c41f8f0600d464399388f01affe8ab7b52bb8031116f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c379b51a78d2042143a423dbcc1c5c8a034c69896cd38fe4f6b726805dc682fc/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f338fea40a56b4b72f7331ecec22e295288fb2173d4041e978bfb254217175bb/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/3d48fa95278b71a9d7b7cb337bb6ccad76721a62a561531b2aa6678b24bfce62/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d747b46c490ebcde468a3c8dbd05500b75e91b3fa870569ff2b5a80d06d9f89d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6ea583fe703846aa6b1d665254d90e9437ff1a1358359410c8deec313a8cae0c/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2f46e93f8b3c748ae404e072baff46d35be75a69b4714c9324e0511c959c209a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/c3eaca3557d68895d0706e6713a7fbf8b7756dff637307407ca924bedab3d8e5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/68d958014d1f7a386ea736eb6274aef3e5ab678af2f9d4db9448f79a13f1d865/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/b51728ee30f2dd7ca88e5e69ea51c76a546316526b36fd285f0dac86c1deb183/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/4cd174b561e411f0ca49e1c25708776c11c4d1900ed2e88bfee8ee04ca45ba69/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/444b30e3cd09dec7486ee677f767683c923d0d1a38e1087714462c6c591e3caf/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/02402b41c9cfdd4c6373579ce97bd150d71dab62efc2b2f71c9e484fa361f7c4/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/a3942bd97e43adf090538ff4f80e1d5637615706978ba641ae2241575f6997a9/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/94ecd6451f4c8cc67ebba70124fa73ebbd4374eee8d11370b8b72fde8ce4e6be/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d3cf3af1eebe8cb4c7f06dd242bdca746019e1db9ad44f7687a1df5c46292859/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/fb50ceb17bcf9ecff9def92c74ec8f8cecf8303d896c8ec6f9a117a963fd505a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/634bb61d05b0788fef739ba1b2335289fc53322600b547be175c0adeb355b7e4/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/23e88012805237f0da5771bab08d51d97671cf840e6ef0549ba2e922be11e53d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7b1a74077d6857d1e7fabd41142ff3a70e3716b951cfe87dd05a35040385c2a2/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/0a2e0131f682d35d52190b1e2603432a38cc99776cd41a599dc070c08d11401a/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/464e9d2600c3134e09de60cb71a65f366fac564eaac6c4ce148cffe177450b1f/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/cc75fab14f367b562a1314533937023bc6d9b89081b8a4f4eabd6dbca63829e0/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/67d842b5cc5e6ce3b6e864e0f687ad680196dcb589346f26c7b27a1bda64f4cc/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/dbefb909c23cf63dd2d2ae1c51833a9e4d3e678bdb9fdae6124651710a265a58/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/d9b02ec6cbab777b1c19b65a9e22cc1f0b4eac8f55973bd1a69cdea2c04cb578/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/d2968cc1a023e62d1f2627f937542a72aab7526a6ba0a529101499bf0c309c1c/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/4eab459bdccb7434f5a422467235781f262d54eaa540a774e7e5c37c0ddd44b6/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/3ab814860f13f9686c3b90d7525f7c5acc4fc5ec7a6c2b456e8196163a84929f/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/0cb6f37f9ec1bdf71a23969d684b05b9e6c0d6232a0f889af64253de5810736b/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/d98c4b1ade1a30364fcf1c6ec235f677c8721c8ea7ba536413c59589186a692a/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/1e013050e3a1123a2bce2e58ee890300c00fb3def1cafd9992d70cba21839de8/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/98fc1e315c62288d8477602650e42b32716ace7c45a78f99fe21984954a0a6ca/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/a539395941b0ba04a75e43cbc22692d055b89c56af5efb4bf0e29bf3dc90b749/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/3c685d1ccdcd58f315b683d8a442d3156b44d309f2ddb55de41fa20704d6dc36/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/7d33665d5dd0913f58151fe90148d3e1319b44d91f12592e8e97cdf1b969f2fb/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/702006f8fce75d192e5e57c0ae6ab3c739e175edd4eb6dc75eeb2675798a090b/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/97f940579aae42a059d76d66e263e7ff5cc70aafbb2d522470d1e2f1f0ed702a/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/84a8f4dca9da59a9d342782142f781283822dacd5ad9826e87e15d30129e7cb8/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/872700a9fc99afa21e360000139ee625c4e28a4a228616ecc56eb0f0f4e082fa/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/07e475ef1f37e7469a8ac33357159ec8c84a36117b3a134b65ddcadcba55574f/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/a4ce81bcf0a573f52bcf7be1e385495430f2e69f112d2ef95a67763e20a1cd5e/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/6f5873ba7ea1b5680568981d9036758161c8598cbba0955e7a9dbbe438b8acb6/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/6d2415ea78474658cffe4ace63eb636fc14e1d665431bdce8a2b5ea39b247e80/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/f25c7abccb368c757c13356d22fbc5692f7e561fa751d7a1e3217604af172d02/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/258f54e2575d840855b1ae1ab0691c5d70d91960f00f05abac1cb36c4862db90/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/3979ffec4895491b09e483f97a35ceaabd509b6b60544d9836434c5ac4ed652e/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/989937d3ef35c4d2a8b4173f26ca81f5eb33ecde0715da73c7d68a90791ee622/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/d732ca0efd01db45a32285cff7fb46f584994f3319043d43e55dcaaf0cd2fdc2/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/b90c0b794b83b6886e593f5668fe97d662bed77fd81049fffd22be14795bfbd9/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/4565cc9b5b12e3162225ce5255a7704dd39b48539f760f5b6ac3cabecb63a2fb/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/9f2c44a158e89771ff2216f0d3b3cc2f392aeabcd95ec6fe7fe58774bb1ee2c6/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/4fd43da1c5bf3be587c4fa299fc201ef9558997deda3fa26835ac6576a848525/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/c06b21ca887e730c22b9d890511941442d3ea5e9b5957450865e01c407ebb8d9/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/3ef8b468916877955f9445bcfc5ca392724e4f706b64ff177b9554459c4658a1/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/b00818c10bff241a462348cff23df1ec8983ebbcfd94802047aaa011bb4683b8/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/4a4ab94f79568019376af68dad8f1e28f4c16c1dbb02655ee528b06d7071b935/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/ca7b4d108656d887fb33043be58c2725be79da95e70eabc816fdac9f3ba785fc/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/d61d6e3b5b340352eb62655b17cab11eb8409441db39a6e7a1657bbfe3e9a318/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/894e93526d0b29837415766afb19352917d1b8a14b3f7a894165b4db1072528e/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/37800adfcee20c57563ce4931c3e7d8476ef7dcb072057a790631da4b075754f/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 19:30:00.219474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.219082    4823 manager.go:217] Machine: {Timestamp:2026-10-30 19:30:00.218567713 +0000 UTC m=+0.111182028 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100156 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2901f6ef5fcc215b05c36c65585c58 SystemUUID:ec2901f6-ef5f-cc21-5b05-c36c65585c58 BootID:b5a1dded-ed9a-45c2-9f47-6c6a545fdd1d Filesystems:[{Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33c7fd5ad315ac4b5b970c81af2a9c37e116079a5627799aa4c7c52b794ed9df/userdata/shm DeviceMajor:0 DeviceMinor:329 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e683558b-f046-46ae-99c4-4862e1247b03/volumes/kubernetes.io~projected/kube-api-access-729fn DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5db63b46-2839-43a5-9d7d-0cf80e8e1aa7/volumes/kubernetes.io~projected/kube-api-access-fdscq DeviceMajor:0 DeviceMinor:341 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6bc162c8-806a-4d1f-a046-3ade90069067/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6ec634f-81f4-4d4e-97ef-33092b38052a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f387d9b5-577c-470c-a25e-da28234f464a/volumes/kubernetes.io~projected/kube-api-access-8s4j8 DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9e0ec109-980a-4203-b45b-25ebd01ae8ed/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbe267911d17fb48905e78682b379f997883866802948fe4cd4f755ef7d8b749/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e683558b-f046-46ae-99c4-4862e1247b03/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/339613c0d5b9ef3b19a2724086dfcd73a391e38e746e7a49802e28c44478c19d/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/468ec5d6-34c5-4577-84a5-ec661a3f2736/volumes/kubernetes.io~projected/kube-api-access-h6qvt DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/39126eba-223c-47c2-aff5-4233035260a3/volumes/kubernetes.io~projected/kube-api-access-7xd7c DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4909e20c-2484-41bd-8f06-67f37dfd337b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12639b64583a28bd0527aef31a1e602431f2b40820fb01411be64e6befa78d5e/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0166d3b7fd00652d2525020761ad62349e7fcd17cb6c5866bd5633d9827af5f5/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28217977-99fe-4540-a322-3b3efb6dfa36/volumes/kubernetes.io~projected/kube-api-access-2bbgk DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6bc162c8-806a-4d1f-a046-3ade90069067/volumes/kubernetes.io~projected/kube-api-access-mp8kb DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/153c1fd1-90a2-4063-b066-6503d796d6d8/volumes/kubernetes.io~projected/kube-api-access-zfdmm DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9e0ec109-980a-4203-b45b-25ebd01ae8ed/volumes/kubernetes.io~projected/kube-api-access-6b8qf DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a9e7044a68634c91220db20574d26df0b9c05fb61bed3060d0df1effb170ace/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/22a7299c-d539-47fa-87e2-626c5ca8bbbf/volumes/kubernetes.io~projected/kube-api-access-g2svq DeviceMajor:0 DeviceMinor:306 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/153c1fd1-90a2-4063-b066-6503d796d6d8/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de4b10692573ed031d00c1d6fcb29945422022be92d25e9d2e6f7c4be83948b9/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f14051b645404afad61dd51236779b08cbf4394d7ec80ee2ceae2f731cd3661d/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b5f201ec-3c4d-4738-b721-ca4e445d63ad/volumes/kubernetes.io~projected/kube-api-access-l6hxs DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da310e19-8c58-4785-a88e-09edc8a63e2e/volumes/kubernetes.io~projected/kube-api-access-4wkt8 DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/392e4212-393d-4aaa-904b-f72f1e287c34/volumes/kubernetes.io~projected/kube-api-access-kn4pl DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b859724c833e248f69a3b21c3a0ed8a9270ce2a8e686d446d0856ad8f0392a2/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7cdfd04130d7c43badbb4acabd61789fcc62d63f0b04bade27b9fcd4876c1cdb/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df3b6a17-4bbf-41dc-83e5-df5172b87b7f/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4909e20c-2484-41bd-8f06-67f37dfd337b/volumes/kubernetes.io~projected/kube-api-access-xjmx9 DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/926e8fa4-1a22-48b1-832e-ea924ef61178/volumes/kubernetes.io~projected/kube-api-access-rzkdg DeviceMajor:0 DeviceMinor:293 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/343d60461be4bd745b9eb08058f432ffb36a1cc7bf0f4822efb0101a0359b6de/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98fe0d24-2073-47c4-af4a-171c54b7eb2d/volumes/kubernetes.io~projected/kube-api-access-99bgq DeviceMajor:0 DeviceMinor:334 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:346 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/153c1fd1-90a2-4063-b066-6503d796d6d8/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/153c1fd1-90a2-4063-b066-6503d796d6d8/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12b4dd3d73c6a1df2d18d4e074867d79373af5546eafec150feaf4aa45bad2ab/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6ec634f-81f4-4d4e-97ef-33092b38052a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c/volumes/kubernetes.io~projected/kube-api-access-cs72w DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c246f08a-6fa3-4664-8a73-567d35f22d3a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac07aa06-d891-4c95-8500-fdef3723ff38/volumes/kubernetes.io~projected/kube-api-access-5vxfk DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0af906a5-4f1e-4feb-8e1d-366cce9964c0/volumes/kubernetes.io~projected/kube-api-access-rzjfz DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e683558b-f046-46ae-99c4-4862e1247b03/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/234270c9f293615bc63d23ff406cb11dc35111515be70d1f31c5a069a2e9f8be/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/159873945b00a7e58f88326c67234600440fdacd52e6967ad11414cc0039e831/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f1402c6a45b330bbb1cc8cd1e011283fcbb6dbc621e3c8679a903e3fa69bbd0/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/468ec5d6-34c5-4577-84a5-ec661a3f2736/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4/volumes/kubernetes.io~projected/kube-api-access-clprx DeviceMajor:0 DeviceMinor:347 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b474aa749d00cbc1c774a8548b3527b813e0931b37dbd84958c4db823745aa5/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf58f46fc376b90dd9b3c55c60be823fca4491f1fe37b32845f586a907dbac35/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7bdcfe753eeaad93d91080684857987cc8d9e8813f718e927802a0681d5902b/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52efc11e-b616-4c33-97d8-06fed7d75f82/volumes/kubernetes.io~projected/kube-api-access-pcfz2 DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bb3df8dc-d134-4cee-b95e-21e32f1df218/volumes/kubernetes.io~projected/kube-api-access-8zgxv DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e0ec109-980a-4203-b45b-25ebd01ae8ed/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b5ff1aae-17b6-4563-949b-e8321be54e25/volumes/kubernetes.io~projected/kube-api-access-jsl6n DeviceMajor:0 DeviceMinor:318 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7a16614b766a578de04bc5f01c9de025ad846dc68802f7bae453eef46f936e8/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89878ff87d7d6936af6e057d953f2858ab03f66e9789b28558a113c376fe20d5/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/281a4c4c3da56a00265e16552431c0542ef51e446b47c275f429272f9e2ba040/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6ec634f-81f4-4d4e-97ef-33092b38052a/volumes/kubernetes.io~projected/kube-api-access-22dvx DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7427b4f7-fba8-4cde-ad90-1f46d5c92d87/volumes/kubernetes.io~projected/kube-api-access-cb4cf DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3615ac3b-837c-46ea-b575-4cdc1edc78a0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/df3b6a17-4bbf-41dc-83e5-df5172b87b7f/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4c7253d-c52b-4f85-bc06-cfafe6443bb4/volumes/kubernetes.io~projected/kube-api-access-gtz66 DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3615ac3b-837c-46ea-b575-4cdc1edc78a0/volumes/kubernetes.io~projected/kube-api-access-jhbv6 DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8b94627ee953e7c239fc66fbf71993f6c710c9eacdb2763ca70ef6aaefc809f/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f87225c4e0d6745d6509a153c22157103d8b00e9661b92b940d95e3fa9abbf23/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df3b6a17-4bbf-41dc-83e5-df5172b87b7f/volumes/kubernetes.io~projected/kube-api-access-5q76f DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24eb387a935fbe718617009ea1258f85c1430143dda2b1b9ac47b8f4e16d4e35/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/ebaeaca4-4212-4bcc-b979-873495265bad/volumes/kubernetes.io~projected/kube-api-access-72lkw DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14ac690ea497ca589a3c1266822bb4e2e626396b5b84648b4d25e101985876f9/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7941072e-d482-42dc-a44e-6e0f9118912f/volumes/kubernetes.io~projected/kube-api-access-884tr DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5/volumes/kubernetes.io~projected/kube-api-access-gkz6g DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17bebd71442cee57df0a63815d5d2ae487a270b957f1bfe226ce7d9475b88fe3/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/392e4212-393d-4aaa-904b-f72f1e287c34/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9e0ec109-980a-4203-b45b-25ebd01ae8ed/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0b859724c833e24 MacAddress:06:27:33:d9:e9:f6 Speed:10000 Mtu:8901} {Name:12639b64583a28b MacAddress:42:8c:02:30:54:3c Speed:10000 Mtu:8901} {Name:12b4dd3d73c6a1d MacAddress:2e:90:98:27:10:7a Speed:10000 Mtu:8901} {Name:159873945b00a7e MacAddress:92:1a:71:0e:b0:aa Speed:10000 Mtu:8901} {Name:1b474aa749d00cb MacAddress:3e:40:77:56:be:3e Speed:10000 Mtu:8901} {Name:234270c9f293615 MacAddress:26:3b:f2:37:06:b3 Speed:10000 Mtu:8901} {Name:24eb387a935fbe7 MacAddress:02:6b:f0:27:df:5b Speed:10000 Mtu:8901} {Name:281a4c4c3da56a0 MacAddress:a2:6b:ec:3e:8d:45 Speed:10000 Mtu:8901} {Name:33c7fd5ad315ac4 MacAddress:0a:28:5e:b2:89:eb Speed:10000 Mtu:8901} {Name:8a9e7044a68634c MacAddress:f2:9d:1a:4c:cc:e9 Speed:10000 Mtu:8901} {Name:bf58f46fc376b90 MacAddress:c6:34:a6:0e:9c:4a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d3:54:57:ed Speed:0 Mtu:9001} {Name:br-int MacAddress:b2:f5:96:c3:01:bf Speed:0 Mtu:8901} {Name:c8b94627ee953e7 MacAddress:9a:22:a9:8e:83:dc Speed:10000 Mtu:8901} {Name:dbe267911d17fb4 MacAddress:d2:10:03:16:c8:48 Speed:10000 Mtu:8901} {Name:de4b10692573ed0 MacAddress:52:12:d0:86:86:b7 Speed:10000 Mtu:8901} {Name:e7bdcfe753eeaad MacAddress:6a:e8:ed:f3:73:63 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d3:54:57:ed Speed:0 Mtu:9001} {Name:f14051b645404af MacAddress:b2:7c:5d:eb:8f:cb Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:fe:2d:30:fa:21:78 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5a:bb:15:d6:9b:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:30:00.219474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.219476    4823 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:30:00.219799 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.219635    4823 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:30:00.219998 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.219972    4823 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:30:00.220141 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.219998    4823 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-61.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:30:00.220183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.220144    4823 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:30:00.220183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.220152    4823 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:30:00.220183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.220166    4823 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:30:00.220183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.220181    4823 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:30:00.220292 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.220286    4823 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:30:00.220506 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.220495    4823 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:30:00.220625 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.220616    4823 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:30:00.220665 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.220657    4823 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:30:00.220694 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.220669    4823 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:30:00.220694 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.220680    4823 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:30:00.220694 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.220689    4823 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:30:00.220799 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.220697    4823 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:30:00.221428 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.221400    4823 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:30:00.221428 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.221417    4823 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:30:00.222894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.222317    4823 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:30:00.225753 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.225721    4823 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:30:00.225753 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.225742    4823 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:30:00.225956 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.225923    4823 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 19:30:00.226143 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.226127    4823 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:30:00.226205 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.226160    4823 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:30:00.226279 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.226254    4823 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:30:00.226279 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.226267    4823 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:30:00.226279 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.226273    4823 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:30:00.226279 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.226278    4823 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:30:00.226394 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.226285    4823 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:30:00.226394 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.226294    4823 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:30:00.226394 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.226303    4823 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:30:00.226394 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.226309    4823 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:30:00.226394 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.226317    4823 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:30:00.226394 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.226323    4823 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:30:00.226394 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.226336    4823 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:30:00.226394 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.226346    4823 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:30:00.226394 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.226375    4823 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:30:00.226394 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.226384    4823 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:30:00.230151 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.230133    4823 server.go:1263] "Started kubelet"
Oct 30 19:30:00.230366 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.230214    4823 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:30:00.230542 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.230480    4823 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:30:00.230602 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.230565    4823 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:30:00.230940 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.230924    4823 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:30:00.230993 ip-10-0-138-61 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:30:00.231532 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.231514    4823 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.233755    4823 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.234010    4823 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.234096    4823 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:24:20 +0000 UTC" deadline="2028-05-18 02:05:20.042108043 +0000 UTC"
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.234131    4823 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13566h35m19.807980469s"
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:00.234164    4823 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-61.us-east-2.compute.internal\" not found"
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.234213    4823 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.234223    4823 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.234618    4823 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.237558    4823 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.237580    4823 factory.go:55] Registering systemd factory
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.237593    4823 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238113    4823 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238284    4823 factory.go:153] Registering CRI-O factory
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238298    4823 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238317    4823 factory.go:103] Registering Raw factory
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238332    4823 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238566    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebaeaca4-4212-4bcc-b979-873495265bad" volumeName="kubernetes.io/projected/ebaeaca4-4212-4bcc-b979-873495265bad-kube-api-access-72lkw" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238611    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4909e20c-2484-41bd-8f06-67f37dfd337b" volumeName="kubernetes.io/projected/4909e20c-2484-41bd-8f06-67f37dfd337b-kube-api-access-xjmx9" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238632    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7427b4f7-fba8-4cde-ad90-1f46d5c92d87" volumeName="kubernetes.io/configmap/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-telemetry-config" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238646    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="926e8fa4-1a22-48b1-832e-ea924ef61178" volumeName="kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238661    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5ff1aae-17b6-4563-949b-e8321be54e25" volumeName="kubernetes.io/empty-dir/b5ff1aae-17b6-4563-949b-e8321be54e25-data-volume" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238681    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebaeaca4-4212-4bcc-b979-873495265bad" volumeName="kubernetes.io/configmap/ebaeaca4-4212-4bcc-b979-873495265bad-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238694    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac07aa06-d891-4c95-8500-fdef3723ff38" volumeName="kubernetes.io/projected/ac07aa06-d891-4c95-8500-fdef3723ff38-kube-api-access-5vxfk" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238712    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c796373-b082-4069-b5e7-228588cc99da" volumeName="kubernetes.io/configmap/0c796373-b082-4069-b5e7-228588cc99da-nginx-conf" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238730    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="392e4212-393d-4aaa-904b-f72f1e287c34" volumeName="kubernetes.io/projected/392e4212-393d-4aaa-904b-f72f1e287c34-kube-api-access-kn4pl" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238744    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e0ec109-980a-4203-b45b-25ebd01ae8ed" volumeName="kubernetes.io/empty-dir/9e0ec109-980a-4203-b45b-25ebd01ae8ed-ca-trust-extracted" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238761    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e0ec109-980a-4203-b45b-25ebd01ae8ed" volumeName="kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-installation-pull-secrets" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238770    4823 manager.go:319] Starting recovery of all containers
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238775    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac07aa06-d891-4c95-8500-fdef3723ff38" volumeName="kubernetes.io/configmap/ac07aa06-d891-4c95-8500-fdef3723ff38-config-volume" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238793    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5ff1aae-17b6-4563-949b-e8321be54e25" volumeName="kubernetes.io/projected/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-api-access-jsl6n" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238807    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6ec634f-81f4-4d4e-97ef-33092b38052a" volumeName="kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovnkube-config" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238845    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153c1fd1-90a2-4063-b066-6503d796d6d8" volumeName="kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-trusted-ca" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238904    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4909e20c-2484-41bd-8f06-67f37dfd337b" volumeName="kubernetes.io/secret/4909e20c-2484-41bd-8f06-67f37dfd337b-serving-cert" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238928    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f851f52-96a2-40cd-9ff0-ed8282ec1a0c" volumeName="kubernetes.io/projected/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-kube-api-access-cs72w" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238943    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e0ec109-980a-4203-b45b-25ebd01ae8ed" volumeName="kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-certificates" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238956    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e683558b-f046-46ae-99c4-4862e1247b03" volumeName="kubernetes.io/projected/e683558b-f046-46ae-99c4-4862e1247b03-kube-api-access-729fn" seLinuxMountContext=""
Oct 30 19:30:00.238952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238974    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f387d9b5-577c-470c-a25e-da28234f464a" volumeName="kubernetes.io/projected/f387d9b5-577c-470c-a25e-da28234f464a-kube-api-access-8s4j8" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.238991    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153c1fd1-90a2-4063-b066-6503d796d6d8" volumeName="kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-installation-pull-secrets" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239008    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="468ec5d6-34c5-4577-84a5-ec661a3f2736" volumeName="kubernetes.io/projected/468ec5d6-34c5-4577-84a5-ec661a3f2736-kube-api-access-h6qvt" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239048    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bc162c8-806a-4d1f-a046-3ade90069067" volumeName="kubernetes.io/configmap/6bc162c8-806a-4d1f-a046-3ade90069067-service-ca-bundle" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239063    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6ec634f-81f4-4d4e-97ef-33092b38052a" volumeName="kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239082    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6ec634f-81f4-4d4e-97ef-33092b38052a" volumeName="kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-env-overrides" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239107    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e" volumeName="kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239120    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca3d3980-29e3-4afa-a9f8-302f62fb0ac4" volumeName="kubernetes.io/projected/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-kube-api-access-clprx" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239138    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3615ac3b-837c-46ea-b575-4cdc1edc78a0" volumeName="kubernetes.io/projected/3615ac3b-837c-46ea-b575-4cdc1edc78a0-kube-api-access-jhbv6" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239171    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb3df8dc-d134-4cee-b95e-21e32f1df218" volumeName="kubernetes.io/projected/bb3df8dc-d134-4cee-b95e-21e32f1df218-kube-api-access-8zgxv" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239186    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da310e19-8c58-4785-a88e-09edc8a63e2e" volumeName="kubernetes.io/configmap/da310e19-8c58-4785-a88e-09edc8a63e2e-serviceca" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239204    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153c1fd1-90a2-4063-b066-6503d796d6d8" volumeName="kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-bound-sa-token" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239219    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153c1fd1-90a2-4063-b066-6503d796d6d8" volumeName="kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239231    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52efc11e-b616-4c33-97d8-06fed7d75f82" volumeName="kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239249    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bc162c8-806a-4d1f-a046-3ade90069067" volumeName="kubernetes.io/empty-dir/6bc162c8-806a-4d1f-a046-3ade90069067-snapshots" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239261    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c246f08a-6fa3-4664-8a73-567d35f22d3a" volumeName="kubernetes.io/configmap/c246f08a-6fa3-4664-8a73-567d35f22d3a-konnectivity-ca" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239276    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4c7253d-c52b-4f85-bc06-cfafe6443bb4" volumeName="kubernetes.io/projected/d4c7253d-c52b-4f85-bc06-cfafe6443bb4-kube-api-access-gtz66" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239290    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5" volumeName="kubernetes.io/projected/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-kube-api-access-gkz6g" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239306    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df3b6a17-4bbf-41dc-83e5-df5172b87b7f" volumeName="kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-stats-auth" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239328    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22a7299c-d539-47fa-87e2-626c5ca8bbbf" volumeName="kubernetes.io/projected/22a7299c-d539-47fa-87e2-626c5ca8bbbf-kube-api-access-g2svq" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239346    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98fe0d24-2073-47c4-af4a-171c54b7eb2d" volumeName="kubernetes.io/projected/98fe0d24-2073-47c4-af4a-171c54b7eb2d-kube-api-access-99bgq" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239370    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6ec634f-81f4-4d4e-97ef-33092b38052a" volumeName="kubernetes.io/projected/c6ec634f-81f4-4d4e-97ef-33092b38052a-kube-api-access-22dvx" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239388    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e683558b-f046-46ae-99c4-4862e1247b03" volumeName="kubernetes.io/empty-dir/e683558b-f046-46ae-99c4-4862e1247b03-tmp" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239404    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebaeaca4-4212-4bcc-b979-873495265bad" volumeName="kubernetes.io/configmap/ebaeaca4-4212-4bcc-b979-873495265bad-multus-daemon-config" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239428    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22a7299c-d539-47fa-87e2-626c5ca8bbbf" volumeName="kubernetes.io/empty-dir/22a7299c-d539-47fa-87e2-626c5ca8bbbf-archives-path" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239445    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="392e4212-393d-4aaa-904b-f72f1e287c34" volumeName="kubernetes.io/configmap/392e4212-393d-4aaa-904b-f72f1e287c34-trusted-ca" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239468    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="468ec5d6-34c5-4577-84a5-ec661a3f2736" volumeName="kubernetes.io/secret/468ec5d6-34c5-4577-84a5-ec661a3f2736-serving-cert" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239486    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bc162c8-806a-4d1f-a046-3ade90069067" volumeName="kubernetes.io/configmap/6bc162c8-806a-4d1f-a046-3ade90069067-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239494    4823 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239504    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c246f08a-6fa3-4664-8a73-567d35f22d3a" volumeName="kubernetes.io/secret/c246f08a-6fa3-4664-8a73-567d35f22d3a-agent-certs" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239527    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6ec634f-81f4-4d4e-97ef-33092b38052a" volumeName="kubernetes.io/secret/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239547    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca3d3980-29e3-4afa-a9f8-302f62fb0ac4" volumeName="kubernetes.io/secret/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-signing-key" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239569    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da310e19-8c58-4785-a88e-09edc8a63e2e" volumeName="kubernetes.io/projected/da310e19-8c58-4785-a88e-09edc8a63e2e-kube-api-access-4wkt8" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239588    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153c1fd1-90a2-4063-b066-6503d796d6d8" volumeName="kubernetes.io/empty-dir/153c1fd1-90a2-4063-b066-6503d796d6d8-ca-trust-extracted" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239609    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52efc11e-b616-4c33-97d8-06fed7d75f82" volumeName="kubernetes.io/projected/52efc11e-b616-4c33-97d8-06fed7d75f82-kube-api-access-pcfz2" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239632    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bc162c8-806a-4d1f-a046-3ade90069067" volumeName="kubernetes.io/empty-dir/6bc162c8-806a-4d1f-a046-3ade90069067-tmp" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239653    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e0ec109-980a-4203-b45b-25ebd01ae8ed" volumeName="kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-bound-sa-token" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239708    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e0ec109-980a-4203-b45b-25ebd01ae8ed" volumeName="kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239749    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df3b6a17-4bbf-41dc-83e5-df5172b87b7f" volumeName="kubernetes.io/projected/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-kube-api-access-5q76f" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239763    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f387d9b5-577c-470c-a25e-da28234f464a" volumeName="kubernetes.io/configmap/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-config" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239782    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0af906a5-4f1e-4feb-8e1d-366cce9964c0" volumeName="kubernetes.io/projected/0af906a5-4f1e-4feb-8e1d-366cce9964c0-kube-api-access-rzjfz" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239796    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="468ec5d6-34c5-4577-84a5-ec661a3f2736" volumeName="kubernetes.io/configmap/468ec5d6-34c5-4577-84a5-ec661a3f2736-config" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239814    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac07aa06-d891-4c95-8500-fdef3723ff38" volumeName="kubernetes.io/empty-dir/ac07aa06-d891-4c95-8500-fdef3723ff38-tmp-dir" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239828    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22a7299c-d539-47fa-87e2-626c5ca8bbbf" volumeName="kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239842    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39126eba-223c-47c2-aff5-4233035260a3" volumeName="kubernetes.io/projected/39126eba-223c-47c2-aff5-4233035260a3-kube-api-access-7xd7c" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239876    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e0ec109-980a-4203-b45b-25ebd01ae8ed" volumeName="kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-kube-api-access-6b8qf" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239890    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5f201ec-3c4d-4738-b721-ca4e445d63ad" volumeName="kubernetes.io/projected/b5f201ec-3c4d-4738-b721-ca4e445d63ad-kube-api-access-l6hxs" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239908    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca3d3980-29e3-4afa-a9f8-302f62fb0ac4" volumeName="kubernetes.io/configmap/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-signing-cabundle" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239922    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7941072e-d482-42dc-a44e-6e0f9118912f" volumeName="kubernetes.io/projected/7941072e-d482-42dc-a44e-6e0f9118912f-kube-api-access-884tr" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239936    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5f201ec-3c4d-4738-b721-ca4e445d63ad" volumeName="kubernetes.io/empty-dir/b5f201ec-3c4d-4738-b721-ca4e445d63ad-tmp-dir" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239953    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153c1fd1-90a2-4063-b066-6503d796d6d8" volumeName="kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-certificates" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239966    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28217977-99fe-4540-a322-3b3efb6dfa36" volumeName="kubernetes.io/projected/28217977-99fe-4540-a322-3b3efb6dfa36-kube-api-access-2bbgk" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239983    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="392e4212-393d-4aaa-904b-f72f1e287c34" volumeName="kubernetes.io/secret/392e4212-393d-4aaa-904b-f72f1e287c34-serving-cert" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.239997    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52efc11e-b616-4c33-97d8-06fed7d75f82" volumeName="kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.240010    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52efc11e-b616-4c33-97d8-06fed7d75f82" volumeName="kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.240029    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5db63b46-2839-43a5-9d7d-0cf80e8e1aa7" volumeName="kubernetes.io/projected/5db63b46-2839-43a5-9d7d-0cf80e8e1aa7-kube-api-access-fdscq" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.240042    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bc162c8-806a-4d1f-a046-3ade90069067" volumeName="kubernetes.io/projected/6bc162c8-806a-4d1f-a046-3ade90069067-kube-api-access-mp8kb" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.240061    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153c1fd1-90a2-4063-b066-6503d796d6d8" volumeName="kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-kube-api-access-zfdmm" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.240075    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bc162c8-806a-4d1f-a046-3ade90069067" volumeName="kubernetes.io/secret/6bc162c8-806a-4d1f-a046-3ade90069067-serving-cert" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.240089    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7427b4f7-fba8-4cde-ad90-1f46d5c92d87" volumeName="kubernetes.io/projected/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-kube-api-access-cb4cf" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.240108    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e0ec109-980a-4203-b45b-25ebd01ae8ed" volumeName="kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-trusted-ca" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.240123    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5" volumeName="kubernetes.io/configmap/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-iptables-alerter-script" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.240140    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df3b6a17-4bbf-41dc-83e5-df5172b87b7f" volumeName="kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-default-certificate" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.240157    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4909e20c-2484-41bd-8f06-67f37dfd337b" volumeName="kubernetes.io/configmap/4909e20c-2484-41bd-8f06-67f37dfd337b-config" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.240171    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22a7299c-d539-47fa-87e2-626c5ca8bbbf" volumeName="kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-service-ca-bundle" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.240193    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3615ac3b-837c-46ea-b575-4cdc1edc78a0" volumeName="kubernetes.io/secret/3615ac3b-837c-46ea-b575-4cdc1edc78a0-cert" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.240209    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="392e4212-393d-4aaa-904b-f72f1e287c34" volumeName="kubernetes.io/configmap/392e4212-393d-4aaa-904b-f72f1e287c34-config" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.240227    4823 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e683558b-f046-46ae-99c4-4862e1247b03" volumeName="kubernetes.io/empty-dir/e683558b-f046-46ae-99c4-4862e1247b03-etc-tuned" seLinuxMountContext=""
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.240238    4823 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:30:00.240465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.240247    4823 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:30:00.242578 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.241171    4823 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:30:00.245526 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.243286    4823 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 19:30:00.259281 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.259255    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 19:30:00.260588 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.260567    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 19:30:00.268613 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.268560    4823 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:30:00.270300 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.270271    4823 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:30:00.270300 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.270298    4823 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:30:00.270490 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.270321    4823 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:30:00.270490 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:00.270362    4823 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 19:30:00.273762 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.273730    4823 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:30:00.290601 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.290571    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 19:30:00.290759 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.290609    4823 generic.go:356] "Generic (PLEG): container finished" podID="392e4212-393d-4aaa-904b-f72f1e287c34" containerID="7faa03714fb97f461d9afe0d24129275695c5969490602c53524e726085fcc81" exitCode=255
Oct 30 19:30:00.303774 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.303740    4823 generic.go:356] "Generic (PLEG): container finished" podID="52efc11e-b616-4c33-97d8-06fed7d75f82" containerID="ca51af1c99614bf904f82b0edc64206e87412719caf2cbd878749057c78a3357" exitCode=0
Oct 30 19:30:00.303774 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.303772    4823 generic.go:356] "Generic (PLEG): container finished" podID="52efc11e-b616-4c33-97d8-06fed7d75f82" containerID="c3963735a2d43d376259775b214a31cedebd80ccb6b7bc6bd0b62aaabc65fc62" exitCode=0
Oct 30 19:30:00.303983 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.303786    4823 generic.go:356] "Generic (PLEG): container finished" podID="52efc11e-b616-4c33-97d8-06fed7d75f82" containerID="1968fe429b56d461a8f4b20d4f06923d4a6a168d63b2c84a49ed69c4572ee2cd" exitCode=0
Oct 30 19:30:00.303983 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.303796    4823 generic.go:356] "Generic (PLEG): container finished" podID="52efc11e-b616-4c33-97d8-06fed7d75f82" containerID="dc1cea409805840974e4cbbf48f28fd2632ab8388baeb4ec2238ce3ca7fe579e" exitCode=0
Oct 30 19:30:00.303983 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.303806    4823 generic.go:356] "Generic (PLEG): container finished" podID="52efc11e-b616-4c33-97d8-06fed7d75f82" containerID="12aef3ec85422a735bd9e555ad20b02cd2ae7a82869b0afe7cf7c4b7eeea51a5" exitCode=0
Oct 30 19:30:00.303983 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.303818    4823 generic.go:356] "Generic (PLEG): container finished" podID="52efc11e-b616-4c33-97d8-06fed7d75f82" containerID="c09c9f078176b179f93efd4d733d052b91d36929bb828c32c63507031f6fa9ed" exitCode=0
Oct 30 19:30:00.307342 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.307311    4823 generic.go:356] "Generic (PLEG): container finished" podID="caaa86be00fc4b40c2a164ffe289a741" containerID="892abd471688048e00426fbf286e327f6b79de36e04178ae2fa217ef29dd98e6" exitCode=0
Oct 30 19:30:00.346399 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.346373    4823 manager.go:324] Recovery completed
Oct 30 19:30:00.370688 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:00.370647    4823 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 19:30:00.371287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.371267    4823 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:30:00.371380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.371315    4823 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:30:00.371380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.371343    4823 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:30:00.371755 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.371728    4823 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 19:30:00.371836 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.371752    4823 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 19:30:00.371836 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.371776    4823 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 19:30:00.371836 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.371784    4823 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 19:30:00.371836 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.371791    4823 policy_none.go:50] "Start"
Oct 30 19:30:00.371836 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.371803    4823 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:30:00.371836 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.371816    4823 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:30:00.380434 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.380405    4823 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 19:30:00.380434 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.380435    4823 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 19:30:00.380567 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.380443    4823 policy_none.go:44] "Start"
Oct 30 19:30:00.385951 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.385907    4823 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:30:00.388699 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.388683    4823 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:30:00.389219 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:00.388710    4823 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:30:00.389219 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.389220    4823 server.go:87] "Starting device plugin registration server"
Oct 30 19:30:00.389483 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.389473    4823 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:30:00.389515 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.389485    4823 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:30:00.389594 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.389575    4823 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:30:00.389712 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.389679    4823 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:30:00.389712 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.389716    4823 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:30:00.390046 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.389721    4823 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:30:00.390046 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.389761    4823 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:30:00.389706258Z","UUID":"9469a4c6-2d93-49eb-a4de-b93e784b1783","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:30:00.391219 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.391199    4823 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:30:00.391317 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.391239    4823 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:30:00.391701 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:00.391671    4823 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:30:00.490020 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.489899    4823 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:30:00.491536 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.491506    4823 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:30:00.491662 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.491543    4823 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:30:00.491662 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.491555    4823 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-61.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:30:00.491662 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.491602    4823 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:00.503635 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.503611    4823 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:00.503801 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.503791    4823 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:00.571062 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.571006    4823 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal"]
Oct 30 19:30:00.571755 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.571700    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal" event={"ID":"f8f0f705095219e55fd87172f433b51c","Type":"ContainerStarted","Data":"6dc3984086e19e0242c772c73e7850ffe588ce8ddf18e52e7ea205a9fa942ee1"}
Oct 30 19:30:00.571897 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.571760    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal" event={"ID":"f8f0f705095219e55fd87172f433b51c","Type":"ContainerStarted","Data":"5f1402c6a45b330bbb1cc8cd1e011283fcbb6dbc621e3c8679a903e3fa69bbd0"}
Oct 30 19:30:00.571897 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.571781    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal" event={"ID":"caaa86be00fc4b40c2a164ffe289a741","Type":"ContainerStarted","Data":"edd8040984c746349d0e39f87860025a1d3288332af63fdf436bdcf7d09f100b"}
Oct 30 19:30:00.571897 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.571801    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal" event={"ID":"caaa86be00fc4b40c2a164ffe289a741","Type":"ContainerDied","Data":"892abd471688048e00426fbf286e327f6b79de36e04178ae2fa217ef29dd98e6"}
Oct 30 19:30:00.571897 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.571827    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal" event={"ID":"caaa86be00fc4b40c2a164ffe289a741","Type":"ContainerStarted","Data":"a7a16614b766a578de04bc5f01c9de025ad846dc68802f7bae453eef46f936e8"}
Oct 30 19:30:00.571897 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.571851    4823 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:00.571897 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.571801    4823 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:00.583737 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.583709    4823 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:30:00.583896 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:00.583762    4823 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:00.584303 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.584288    4823 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:30:00.584347 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:00.584324    4823 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:00.644174 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.644131    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caaa86be00fc4b40c2a164ffe289a741-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"caaa86be00fc4b40c2a164ffe289a741\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:00.644174 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.644174    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8f0f705095219e55fd87172f433b51c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"f8f0f705095219e55fd87172f433b51c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:00.644393 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.644191    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/caaa86be00fc4b40c2a164ffe289a741-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"caaa86be00fc4b40c2a164ffe289a741\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:00.744839 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.744748    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/caaa86be00fc4b40c2a164ffe289a741-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"caaa86be00fc4b40c2a164ffe289a741\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:00.744839 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.744789    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caaa86be00fc4b40c2a164ffe289a741-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"caaa86be00fc4b40c2a164ffe289a741\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:00.744839 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.744805    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8f0f705095219e55fd87172f433b51c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"f8f0f705095219e55fd87172f433b51c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:00.745053 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.744907    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/caaa86be00fc4b40c2a164ffe289a741-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"caaa86be00fc4b40c2a164ffe289a741\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:00.745053 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.744930    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8f0f705095219e55fd87172f433b51c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"f8f0f705095219e55fd87172f433b51c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:00.745053 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:00.744973    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caaa86be00fc4b40c2a164ffe289a741-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal\" (UID: \"caaa86be00fc4b40c2a164ffe289a741\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:01.221248 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.221209    4823 apiserver.go:52] "Watching apiserver"
Oct 30 19:30:01.230347 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.230321    4823 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:30:01.231196 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.231165    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v9m7c","openshift-image-registry/node-ca-lcxk5","openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv","openshift-image-registry/image-registry-5fdc8795f9-cfh7p","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw","kube-system/kas-connection-checker-685895c7c9-ntrs2","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hv5j2","openshift-ingress/router-default-6cb4fdcd5-zsxtw","openshift-network-console/networking-console-plugin-86c478bb88-bwt58","openshift-network-operator/iptables-alerter-f2s9d","openshift-image-registry/image-registry-7ff684fc5d-nksx9","openshift-ovn-kubernetes/ovnkube-node-d4k5h","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk","kube-system/konnectivity-agent-ncbz9","openshift-kube-storage-version-migrator/migrator-66df5d496b-2p5ts","openshift-network-diagnostics/network-check-source-6f78d59f99-j6dm5","openshift-multus/multus-k6rd4","openshift-dns/node-resolver-2sdcp","kube-system/global-pull-secret-syncer-4zdqg","openshift-multus/multus-additional-cni-plugins-wg8f2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7","openshift-kube-storage-version-migrator/migrator-66df5d496b-7k4fp","openshift-service-ca/service-ca-5fffc97bdf-4h84r","openshift-insights/insights-runtime-extractor-8nh5p","kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-fcz7q","openshift-dns/dns-default-kl9jq","openshift-insights/periodic-gathering-pfjv8-8n5cz","openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8","openshift-network-diagnostics/network-check-target-6wwsm","openshift-insights/insights-operator-7c4986d449-4vrds","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal","kube-system/kas-connection-checker-685895c7c9-qjqz6","openshift-console-operator/console-operator-7cd56b8dbc-dp8wx","openshift-ingress-canary/ingress-canary-vzpfb","openshift-multus/network-metrics-daemon-87b4p"]
Oct 30 19:30:01.233166 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.233139    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:30:01.233578 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.233540    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:30:01.235355 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.235330    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:01.235750 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.235712    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 19:30:01.236301 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.236273    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.236413 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.236394    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rhpjd\""
Oct 30 19:30:01.236494 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.236474    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j9zzn\""
Oct 30 19:30:01.237730 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.237714    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:30:01.239420 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.239392    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 19:30:01.239639 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.239617    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hpls5\""
Oct 30 19:30:01.239694 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.239627    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 19:30:01.240496 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.240475    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 19:30:01.241375 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.241355    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 19:30:01.241454 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.241430    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bkswr\""
Oct 30 19:30:01.241667 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.241651    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 19:30:01.241734 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.241696    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 19:30:01.242064 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.242040    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-s4jds\""
Oct 30 19:30:01.242266 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.242250    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pk7z8\""
Oct 30 19:30:01.242337 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.242324    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-psc6g\""
Oct 30 19:30:01.242929 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.242909    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.243583 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.243564    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.243685 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.243671    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 19:30:01.244000 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.243979    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.244184 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.244164    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xgmwc\""
Oct 30 19:30:01.244273 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.244255    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p4b2c\""
Oct 30 19:30:01.244382 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.244355    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 19:30:01.244541 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.244527    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 19:30:01.244610 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.244556    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.244801 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.244783    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 19:30:01.245008 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.244990    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.245072 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.245013    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.245311 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.245293    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-8ztmt\""
Oct 30 19:30:01.246891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.246817    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.247004 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.246904    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 19:30:01.247004 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.246924    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.247004 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.246956    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2prmp\""
Oct 30 19:30:01.247004 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.246987    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:30:01.247179 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247027    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.247179 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247026    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 19:30:01.247179 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247066    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 19:30:01.247179 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247097    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 19:30:01.247179 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247111    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.247179 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247147    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.247459 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247219    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6bc162c8-806a-4d1f-a046-3ade90069067-snapshots\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:30:01.247459 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247248    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc162c8-806a-4d1f-a046-3ade90069067-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:30:01.247459 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247279    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:30:01.247459 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247304    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/468ec5d6-34c5-4577-84a5-ec661a3f2736-serving-cert\") pod \"service-ca-operator-544d749c8c-d2pkv\" (UID: \"468ec5d6-34c5-4577-84a5-ec661a3f2736\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:30:01.247459 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247329    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/392e4212-393d-4aaa-904b-f72f1e287c34-trusted-ca\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:30:01.247459 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247353    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn4pl\" (UniqueName: \"kubernetes.io/projected/392e4212-393d-4aaa-904b-f72f1e287c34-kube-api-access-kn4pl\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:30:01.247459 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247385    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xd7c\" (UniqueName: \"kubernetes.io/projected/39126eba-223c-47c2-aff5-4233035260a3-kube-api-access-7xd7c\") pod \"kas-connection-checker-685895c7c9-ntrs2\" (UID: \"39126eba-223c-47c2-aff5-4233035260a3\") " pod="kube-system/kas-connection-checker-685895c7c9-ntrs2"
Oct 30 19:30:01.247459 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247409    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc162c8-806a-4d1f-a046-3ade90069067-service-ca-bundle\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:30:01.247459 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247414    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:30:01.247459 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247433    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:30:01.247459 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247458    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb4cf\" (UniqueName: \"kubernetes.io/projected/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-kube-api-access-cb4cf\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:30:01.248071 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247480    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/468ec5d6-34c5-4577-84a5-ec661a3f2736-config\") pod \"service-ca-operator-544d749c8c-d2pkv\" (UID: \"468ec5d6-34c5-4577-84a5-ec661a3f2736\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:30:01.248071 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247503    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6qvt\" (UniqueName: \"kubernetes.io/projected/468ec5d6-34c5-4577-84a5-ec661a3f2736-kube-api-access-h6qvt\") pod \"service-ca-operator-544d749c8c-d2pkv\" (UID: \"468ec5d6-34c5-4577-84a5-ec661a3f2736\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:30:01.248071 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247527    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzjfz\" (UniqueName: \"kubernetes.io/projected/0af906a5-4f1e-4feb-8e1d-366cce9964c0-kube-api-access-rzjfz\") pod \"kas-connection-checker-685895c7c9-qjqz6\" (UID: \"0af906a5-4f1e-4feb-8e1d-366cce9964c0\") " pod="kube-system/kas-connection-checker-685895c7c9-qjqz6"
Oct 30 19:30:01.248071 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247549    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6bc162c8-806a-4d1f-a046-3ade90069067-serving-cert\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:30:01.248071 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247575    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp8kb\" (UniqueName: \"kubernetes.io/projected/6bc162c8-806a-4d1f-a046-3ade90069067-kube-api-access-mp8kb\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:30:01.248071 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247601    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtz66\" (UniqueName: \"kubernetes.io/projected/d4c7253d-c52b-4f85-bc06-cfafe6443bb4-kube-api-access-gtz66\") pod \"volume-data-source-validator-58c9f5f46d-hv5j2\" (UID: \"d4c7253d-c52b-4f85-bc06-cfafe6443bb4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hv5j2"
Oct 30 19:30:01.248071 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247625    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s4j8\" (UniqueName: \"kubernetes.io/projected/f387d9b5-577c-470c-a25e-da28234f464a-kube-api-access-8s4j8\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:30:01.248071 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247650    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-884tr\" (UniqueName: \"kubernetes.io/projected/7941072e-d482-42dc-a44e-6e0f9118912f-kube-api-access-884tr\") pod \"network-check-source-6f78d59f99-j6dm5\" (UID: \"7941072e-d482-42dc-a44e-6e0f9118912f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j6dm5"
Oct 30 19:30:01.248071 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247893    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.248071 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247974    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc162c8-806a-4d1f-a046-3ade90069067-service-ca-bundle\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:30:01.248071 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248015    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.248071 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248062    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248119    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6bc162c8-806a-4d1f-a046-3ade90069067-serving-cert\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248172    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/392e4212-393d-4aaa-904b-f72f1e287c34-config\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248203    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/392e4212-393d-4aaa-904b-f72f1e287c34-serving-cert\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248236    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bbgk\" (UniqueName: \"kubernetes.io/projected/28217977-99fe-4540-a322-3b3efb6dfa36-kube-api-access-2bbgk\") pod \"kas-connection-checker-685895c7c9-v9m7c\" (UID: \"28217977-99fe-4540-a322-3b3efb6dfa36\") " pod="kube-system/kas-connection-checker-685895c7c9-v9m7c"
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248265    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bc162c8-806a-4d1f-a046-3ade90069067-tmp\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248319    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248350    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6bc162c8-806a-4d1f-a046-3ade90069067-snapshots\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248354    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248382    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/468ec5d6-34c5-4577-84a5-ec661a3f2736-config\") pod \"service-ca-operator-544d749c8c-d2pkv\" (UID: \"468ec5d6-34c5-4577-84a5-ec661a3f2736\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248436    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.247977    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248502    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/468ec5d6-34c5-4577-84a5-ec661a3f2736-serving-cert\") pod \"service-ca-operator-544d749c8c-d2pkv\" (UID: \"468ec5d6-34c5-4577-84a5-ec661a3f2736\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248529    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248558    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248580    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q2kw5\""
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248604    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:30:01.248618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248629    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.249365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248578    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:30:01.249365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248685    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.249365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248692    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/392e4212-393d-4aaa-904b-f72f1e287c34-serving-cert\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:30:01.249365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248718    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.249365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248760    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.249365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248774    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bc162c8-806a-4d1f-a046-3ade90069067-tmp\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:30:01.249365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248793    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/392e4212-393d-4aaa-904b-f72f1e287c34-config\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:30:01.249365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248804    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.249365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248849    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 19:30:01.249365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248924    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.249365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.248997    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 19:30:01.249365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.249162    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:30:01.250006 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.249396    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.250006 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.249441    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.250006 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.249462    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bvpxn\""
Oct 30 19:30:01.250006 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.249565    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 19:30:01.250006 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.249583    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 19:30:01.250006 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.249618    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:30:01.250006 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.249647    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:30:01.250006 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.249660    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.250006 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.249691    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.250006 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.249771    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fbg5p\""
Oct 30 19:30:01.250006 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.249940    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 19:30:01.251582 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.251563    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:30:01.252643 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.252622    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.252777 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.252759    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:30:01.252837 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.252786    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:30:01.252837 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.252798    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.252927 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.252842    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wp4kx\""
Oct 30 19:30:01.252965 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.252930    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.252965 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.252961    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p6d49\""
Oct 30 19:30:01.253087 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.253066    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:30:01.253187 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.253104    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:30:01.253187 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.253139    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pn76m\""
Oct 30 19:30:01.253666 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.253652    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:30:01.254388 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.254367    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:30:01.254490 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.254405    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.254490 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.254438    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:30:01.254672 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.254656    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.254744 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.254727    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.254796 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.254765    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:30:01.254892 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.254846    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.254952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.254924    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xs4jl\""
Oct 30 19:30:01.254952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.254930    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2cs46\""
Oct 30 19:30:01.255064 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.254963    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8hwq2\""
Oct 30 19:30:01.257430 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.257406    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:30:01.257557 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.257543    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:30:01.257656 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.257642    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:30:01.257769 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.257753    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.257840 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.257754    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zqbbj\""
Oct 30 19:30:01.258031 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.258013    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-q9kwd\""
Oct 30 19:30:01.258167 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.258141    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:01.258393 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.258376    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:30:01.258609 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.258594    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z724k\""
Oct 30 19:30:01.258700 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.258661    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:30:01.259549 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.259522    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bc162c8-806a-4d1f-a046-3ade90069067-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:30:01.259661 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.259637    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/392e4212-393d-4aaa-904b-f72f1e287c34-trusted-ca\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:30:01.259977 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.259955    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xd7c\" (UniqueName: \"kubernetes.io/projected/39126eba-223c-47c2-aff5-4233035260a3-kube-api-access-7xd7c\") pod \"kas-connection-checker-685895c7c9-ntrs2\" (UID: \"39126eba-223c-47c2-aff5-4233035260a3\") " pod="kube-system/kas-connection-checker-685895c7c9-ntrs2"
Oct 30 19:30:01.260633 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.260602    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6qvt\" (UniqueName: \"kubernetes.io/projected/468ec5d6-34c5-4577-84a5-ec661a3f2736-kube-api-access-h6qvt\") pod \"service-ca-operator-544d749c8c-d2pkv\" (UID: \"468ec5d6-34c5-4577-84a5-ec661a3f2736\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv"
Oct 30 19:30:01.261884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.261841    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtz66\" (UniqueName: \"kubernetes.io/projected/d4c7253d-c52b-4f85-bc06-cfafe6443bb4-kube-api-access-gtz66\") pod \"volume-data-source-validator-58c9f5f46d-hv5j2\" (UID: \"d4c7253d-c52b-4f85-bc06-cfafe6443bb4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hv5j2"
Oct 30 19:30:01.262346 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.262325    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp8kb\" (UniqueName: \"kubernetes.io/projected/6bc162c8-806a-4d1f-a046-3ade90069067-kube-api-access-mp8kb\") pod \"insights-operator-7c4986d449-4vrds\" (UID: \"6bc162c8-806a-4d1f-a046-3ade90069067\") " pod="openshift-insights/insights-operator-7c4986d449-4vrds"
Oct 30 19:30:01.262502 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.262484    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:30:01.262565 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.262511    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:30:01.262565 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.262540    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-884tr\" (UniqueName: \"kubernetes.io/projected/7941072e-d482-42dc-a44e-6e0f9118912f-kube-api-access-884tr\") pod \"network-check-source-6f78d59f99-j6dm5\" (UID: \"7941072e-d482-42dc-a44e-6e0f9118912f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j6dm5"
Oct 30 19:30:01.262635 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.262622    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:30:01.262712 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.262692    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.262809 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.262713    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-svfmj\""
Oct 30 19:30:01.262809 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.262711    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.262926 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.262849    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:30:01.262926 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.262893    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.262926 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.262895    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-j4mvd\""
Oct 30 19:30:01.263087 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.262933    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ttn9c\""
Oct 30 19:30:01.263087 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.262992    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn4pl\" (UniqueName: \"kubernetes.io/projected/392e4212-393d-4aaa-904b-f72f1e287c34-kube-api-access-kn4pl\") pod \"console-operator-7cd56b8dbc-dp8wx\" (UID: \"392e4212-393d-4aaa-904b-f72f1e287c34\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:30:01.266533 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.266513    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:30:01.266697 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.266683    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zsvdk\""
Oct 30 19:30:01.266729 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.266713    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.267854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.267832    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzjfz\" (UniqueName: \"kubernetes.io/projected/0af906a5-4f1e-4feb-8e1d-366cce9964c0-kube-api-access-rzjfz\") pod \"kas-connection-checker-685895c7c9-qjqz6\" (UID: \"0af906a5-4f1e-4feb-8e1d-366cce9964c0\") " pod="kube-system/kas-connection-checker-685895c7c9-qjqz6"
Oct 30 19:30:01.268010 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.267991    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bbgk\" (UniqueName: \"kubernetes.io/projected/28217977-99fe-4540-a322-3b3efb6dfa36-kube-api-access-2bbgk\") pod \"kas-connection-checker-685895c7c9-v9m7c\" (UID: \"28217977-99fe-4540-a322-3b3efb6dfa36\") " pod="kube-system/kas-connection-checker-685895c7c9-v9m7c"
Oct 30 19:30:01.268397 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.268377    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb4cf\" (UniqueName: \"kubernetes.io/projected/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-kube-api-access-cb4cf\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:30:01.268590 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.268574    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s4j8\" (UniqueName: \"kubernetes.io/projected/f387d9b5-577c-470c-a25e-da28234f464a-kube-api-access-8s4j8\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:30:01.269085 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.269070    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 19:30:01.269144 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.269098    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.269144 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.269115    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 19:30:01.269144 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.269131    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 19:30:01.269261 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.269131    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xzbzm\""
Oct 30 19:30:01.269261 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.269153    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 19:30:01.337324 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.337286    4823 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:30:01.339583 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.339563    4823 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:01.339695 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.339677    4823 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:01.349171 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349148    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vxfk\" (UniqueName: \"kubernetes.io/projected/ac07aa06-d891-4c95-8500-fdef3723ff38-kube-api-access-5vxfk\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:30:01.349285 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349177    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-trusted-ca\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.349285 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349196    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wkt8\" (UniqueName: \"kubernetes.io/projected/da310e19-8c58-4785-a88e-09edc8a63e2e-kube-api-access-4wkt8\") pod \"node-ca-lcxk5\" (UID: \"da310e19-8c58-4785-a88e-09edc8a63e2e\") " pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:30:01.349285 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349218    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-sys\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.349285 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349242    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-trusted-ca\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.349285 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349265    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-etc-selinux\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.349473 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349452    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:01.349523 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349500    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhbv6\" (UniqueName: \"kubernetes.io/projected/3615ac3b-837c-46ea-b575-4cdc1edc78a0-kube-api-access-jhbv6\") pod \"ingress-canary-vzpfb\" (UID: \"3615ac3b-837c-46ea-b575-4cdc1edc78a0\") " pod="openshift-ingress-canary/ingress-canary-vzpfb"
Oct 30 19:30:01.349567 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349544    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6hxs\" (UniqueName: \"kubernetes.io/projected/b5f201ec-3c4d-4738-b721-ca4e445d63ad-kube-api-access-l6hxs\") pod \"node-resolver-2sdcp\" (UID: \"b5f201ec-3c4d-4738-b721-ca4e445d63ad\") " pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:30:01.349620 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349579    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.349669 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349615    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-trusted-ca\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.349669 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349613    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-ovn\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.349759 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349678    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-image-registry-private-configuration\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.349759 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349722    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.349878 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349824    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-trusted-ca\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.349936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349888    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysctl-conf\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.349936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349919    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2svq\" (UniqueName: \"kubernetes.io/projected/22a7299c-d539-47fa-87e2-626c5ca8bbbf-kube-api-access-g2svq\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:30:01.350015 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349942    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bwt58\" (UID: \"0c796373-b082-4069-b5e7-228588cc99da\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:30:01.350015 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349954    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-image-registry-private-configuration\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.350015 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.349983    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsl6n\" (UniqueName: \"kubernetes.io/projected/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-api-access-jsl6n\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:01.350142 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350075    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjmx9\" (UniqueName: \"kubernetes.io/projected/4909e20c-2484-41bd-8f06-67f37dfd337b-kube-api-access-xjmx9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk\" (UID: \"4909e20c-2484-41bd-8f06-67f37dfd337b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:30:01.350142 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350111    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-node-log\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.350142 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350131    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.350332 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350201    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-socket-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.350332 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350228    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/153c1fd1-90a2-4063-b066-6503d796d6d8-ca-trust-extracted\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.350332 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350248    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-device-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.350332 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350281    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdscq\" (UniqueName: \"kubernetes.io/projected/5db63b46-2839-43a5-9d7d-0cf80e8e1aa7-kube-api-access-fdscq\") pod \"migrator-66df5d496b-7k4fp\" (UID: \"5db63b46-2839-43a5-9d7d-0cf80e8e1aa7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7k4fp"
Oct 30 19:30:01.350332 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350299    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:30:01.350332 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350320    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:30:01.350608 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350330    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/153c1fd1-90a2-4063-b066-6503d796d6d8-ca-trust-extracted\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.350608 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350370    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e0ec109-980a-4203-b45b-25ebd01ae8ed-ca-trust-extracted\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.350608 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350395    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysconfig\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.350608 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.350404    4823 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:30:01.350608 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350432    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-lib-modules\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.350608 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350471    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e0ec109-980a-4203-b45b-25ebd01ae8ed-ca-trust-extracted\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.350608 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350489    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-installation-pull-secrets\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.350608 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350563    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-bound-sa-token\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.350608 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.350597    4823 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls podName:f387d9b5-577c-470c-a25e-da28234f464a nodeName:}" failed. No retries permitted until 2026-10-30 19:30:01.850575109 +0000 UTC m=+1.743189439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xdndw" (UID: "f387d9b5-577c-470c-a25e-da28234f464a") : secret "samples-operator-tls" not found
Oct 30 19:30:01.350993 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350639    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5ff1aae-17b6-4563-949b-e8321be54e25-crio-socket\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:01.350993 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350677    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-installation-pull-secrets\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.350993 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350734    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c796373-b082-4069-b5e7-228588cc99da-nginx-conf\") pod \"networking-console-plugin-86c478bb88-bwt58\" (UID: \"0c796373-b082-4069-b5e7-228588cc99da\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:30:01.350993 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350777    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:01.350993 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350924    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovnkube-script-lib\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.350993 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350959    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-bound-sa-token\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.350993 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.350968    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c796373-b082-4069-b5e7-228588cc99da-nginx-conf\") pod \"networking-console-plugin-86c478bb88-bwt58\" (UID: \"0c796373-b082-4069-b5e7-228588cc99da\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:30:01.351304 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351041    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-k8s-cni-cncf-io\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.351304 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351085    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-trusted-ca-bundle\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:30:01.351304 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351130    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-kubernetes\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.351304 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351158    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-signing-key\") pod \"service-ca-5fffc97bdf-4h84r\" (UID: \"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:30:01.351304 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351227    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.351304 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351277    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-env-overrides\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.351560 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351329    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovnkube-script-lib\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.351560 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351367    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-installation-pull-secrets\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.351560 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351396    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfdmm\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-kube-api-access-zfdmm\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.351560 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351429    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-trusted-ca-bundle\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:30:01.351560 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351433    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-env-overrides\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.351560 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351498    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-signing-key\") pod \"service-ca-5fffc97bdf-4h84r\" (UID: \"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:30:01.351560 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351511    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b5f201ec-3c4d-4738-b721-ca4e445d63ad-hosts-file\") pod \"node-resolver-2sdcp\" (UID: \"b5f201ec-3c4d-4738-b721-ca4e445d63ad\") " pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:30:01.351560 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351542    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22dvx\" (UniqueName: \"kubernetes.io/projected/c6ec634f-81f4-4d4e-97ef-33092b38052a-kube-api-access-22dvx\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.351922 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351567    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-sys-fs\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.351922 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351589    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:30:01.351922 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351591    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-installation-pull-secrets\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.351922 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351635    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:01.351922 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351665    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-systemd\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.351922 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351690    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-run\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.351922 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351716    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-var-lib-kubelet\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.351922 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351785    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4909e20c-2484-41bd-8f06-67f37dfd337b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk\" (UID: \"4909e20c-2484-41bd-8f06-67f37dfd337b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:30:01.351922 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351803    4823 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:30:01.351922 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351822    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-cni-bin\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.351922 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.351847    4823 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:01.351922 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351915    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-stats-auth\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:01.352474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351955    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-run-netns\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.352474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.351962    4823 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:30:01.352474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352022    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-image-registry-private-configuration\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.352474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352099    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4909e20c-2484-41bd-8f06-67f37dfd337b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk\" (UID: \"4909e20c-2484-41bd-8f06-67f37dfd337b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:30:01.352474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352131    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkdg\" (UniqueName: \"kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg\") pod \"network-check-target-6wwsm\" (UID: \"926e8fa4-1a22-48b1-832e-ea924ef61178\") " pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:30:01.352474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352161    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-stats-auth\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:01.352474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352227    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-image-registry-private-configuration\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.352474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352226    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/22a7299c-d539-47fa-87e2-626c5ca8bbbf-archives-path\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:30:01.352474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352250    4823 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:30:01.352474 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.352288    4823 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-61.us-east-2.compute.internal"
Oct 30 19:30:01.352474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352337    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b8qf\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-kube-api-access-6b8qf\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.352474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352361    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/22a7299c-d539-47fa-87e2-626c5ca8bbbf-archives-path\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:30:01.352474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352393    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.353083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352512    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q76f\" (UniqueName: \"kubernetes.io/projected/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-kube-api-access-5q76f\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:01.353083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352541    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-729fn\" (UniqueName: \"kubernetes.io/projected/e683558b-f046-46ae-99c4-4862e1247b03-kube-api-access-729fn\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.353083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352641    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.353083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352691    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.353083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352755    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-netns\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.353083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352783    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysctl-d\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.353083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352810    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-kubelet-config\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:30:01.353083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352837    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-etc-kubernetes\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.353083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352891    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs72w\" (UniqueName: \"kubernetes.io/projected/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-kube-api-access-cs72w\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.353083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352915    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac07aa06-d891-4c95-8500-fdef3723ff38-config-volume\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:30:01.353083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352931    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-cni-binary-copy\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.353083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352947    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcfz2\" (UniqueName: \"kubernetes.io/projected/52efc11e-b616-4c33-97d8-06fed7d75f82-kube-api-access-pcfz2\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.353083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.352990    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c246f08a-6fa3-4664-8a73-567d35f22d3a-konnectivity-ca\") pod \"konnectivity-agent-ncbz9\" (UID: \"c246f08a-6fa3-4664-8a73-567d35f22d3a\") " pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:30:01.353083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353024    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-slash\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353142    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-cni-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353124    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52efc11e-b616-4c33-97d8-06fed7d75f82-cni-binary-copy\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353170    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-hostroot\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353215    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353242    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac07aa06-d891-4c95-8500-fdef3723ff38-config-volume\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353248    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-iptables-alerter-script\") pod \"iptables-alerter-f2s9d\" (UID: \"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5\") " pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353286    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c246f08a-6fa3-4664-8a73-567d35f22d3a-konnectivity-ca\") pod \"konnectivity-agent-ncbz9\" (UID: \"c246f08a-6fa3-4664-8a73-567d35f22d3a\") " pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353289    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5ff1aae-17b6-4563-949b-e8321be54e25-data-volume\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353335    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4909e20c-2484-41bd-8f06-67f37dfd337b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk\" (UID: \"4909e20c-2484-41bd-8f06-67f37dfd337b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353358    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5ff1aae-17b6-4563-949b-e8321be54e25-data-volume\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353362    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-var-lib-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353404    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-kubelet\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353509    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-certificates\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353590    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-systemd\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353592    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-iptables-alerter-script\") pod \"iptables-alerter-f2s9d\" (UID: \"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5\") " pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353642    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e683558b-f046-46ae-99c4-4862e1247b03-etc-tuned\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.353693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353694    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e683558b-f046-46ae-99c4-4862e1247b03-tmp\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353720    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-dbus\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353745    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-etc-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353753    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4909e20c-2484-41bd-8f06-67f37dfd337b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk\" (UID: \"4909e20c-2484-41bd-8f06-67f37dfd337b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353773    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebaeaca4-4212-4bcc-b979-873495265bad-multus-daemon-config\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353815    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-host-slash\") pod \"iptables-alerter-f2s9d\" (UID: \"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5\") " pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353836    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e683558b-f046-46ae-99c4-4862e1247b03-etc-tuned\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353838    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e683558b-f046-46ae-99c4-4862e1247b03-tmp\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353842    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-cnibin\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353918    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da310e19-8c58-4785-a88e-09edc8a63e2e-serviceca\") pod \"node-ca-lcxk5\" (UID: \"da310e19-8c58-4785-a88e-09edc8a63e2e\") " pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.353936    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-modprobe-d\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354123    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ebaeaca4-4212-4bcc-b979-873495265bad-multus-daemon-config\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354154    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-host\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354180    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-system-cni-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354202    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-cni-multus\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354222    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-service-ca-bundle\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354249    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs\") pod \"network-metrics-daemon-87b4p\" (UID: \"bb3df8dc-d134-4cee-b95e-21e32f1df218\") " pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354267    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c246f08a-6fa3-4664-8a73-567d35f22d3a-agent-certs\") pod \"konnectivity-agent-ncbz9\" (UID: \"c246f08a-6fa3-4664-8a73-567d35f22d3a\") " pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354290    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clprx\" (UniqueName: \"kubernetes.io/projected/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-kube-api-access-clprx\") pod \"service-ca-5fffc97bdf-4h84r\" (UID: \"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354289    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-certificates\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354308    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-socket-dir-parent\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354323    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-multus-certs\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354342    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354400    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da310e19-8c58-4785-a88e-09edc8a63e2e-host\") pod \"node-ca-lcxk5\" (UID: \"da310e19-8c58-4785-a88e-09edc8a63e2e\") " pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354421    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99bgq\" (UniqueName: \"kubernetes.io/projected/98fe0d24-2073-47c4-af4a-171c54b7eb2d-kube-api-access-99bgq\") pod \"migrator-66df5d496b-2p5ts\" (UID: \"98fe0d24-2073-47c4-af4a-171c54b7eb2d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2p5ts"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354438    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354457    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-systemd-units\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354456    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-service-ca-bundle\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354474    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-run-ovn-kubernetes\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.354555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354494    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-certificates\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354596    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-cnibin\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354616    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-default-certificate\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354637    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zgxv\" (UniqueName: \"kubernetes.io/projected/bb3df8dc-d134-4cee-b95e-21e32f1df218-kube-api-access-8zgxv\") pod \"network-metrics-daemon-87b4p\" (UID: \"bb3df8dc-d134-4cee-b95e-21e32f1df218\") " pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354661    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-cni-bin\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354686    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-conf-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354735    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5f201ec-3c4d-4738-b721-ca4e445d63ad-tmp-dir\") pod \"node-resolver-2sdcp\" (UID: \"b5f201ec-3c4d-4738-b721-ca4e445d63ad\") " pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354775    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-cni-netd\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354826    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebaeaca4-4212-4bcc-b979-873495265bad-cni-binary-copy\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354906    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354938    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3615ac3b-837c-46ea-b575-4cdc1edc78a0-cert\") pod \"ingress-canary-vzpfb\" (UID: \"3615ac3b-837c-46ea-b575-4cdc1edc78a0\") " pod="openshift-ingress-canary/ingress-canary-vzpfb"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354967    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-default-certificate\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.354967    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5f201ec-3c4d-4738-b721-ca4e445d63ad-tmp-dir\") pod \"node-resolver-2sdcp\" (UID: \"b5f201ec-3c4d-4738-b721-ca4e445d63ad\") " pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355021    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da310e19-8c58-4785-a88e-09edc8a63e2e-serviceca\") pod \"node-ca-lcxk5\" (UID: \"da310e19-8c58-4785-a88e-09edc8a63e2e\") " pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355024    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-kubelet\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355058    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-original-pull-secret\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355075    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovn-node-metrics-cert\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355120    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72lkw\" (UniqueName: \"kubernetes.io/projected/ebaeaca4-4212-4bcc-b979-873495265bad-kube-api-access-72lkw\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355143    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebaeaca4-4212-4bcc-b979-873495265bad-cni-binary-copy\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355151    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-registration-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355236    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkz6g\" (UniqueName: \"kubernetes.io/projected/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-kube-api-access-gkz6g\") pod \"iptables-alerter-f2s9d\" (UID: \"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5\") " pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355240    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovn-node-metrics-cert\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355279    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-system-cni-dir\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355314    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-log-socket\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355339    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355361    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3615ac3b-837c-46ea-b575-4cdc1edc78a0-cert\") pod \"ingress-canary-vzpfb\" (UID: \"3615ac3b-837c-46ea-b575-4cdc1edc78a0\") " pod="openshift-ingress-canary/ingress-canary-vzpfb"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355435    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac07aa06-d891-4c95-8500-fdef3723ff38-tmp-dir\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355470    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-signing-cabundle\") pod \"service-ca-5fffc97bdf-4h84r\" (UID: \"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355482    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c246f08a-6fa3-4664-8a73-567d35f22d3a-agent-certs\") pod \"konnectivity-agent-ncbz9\" (UID: \"c246f08a-6fa3-4664-8a73-567d35f22d3a\") " pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355534    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac07aa06-d891-4c95-8500-fdef3723ff38-tmp-dir\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355511    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovnkube-config\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355679    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-signing-cabundle\") pod \"service-ca-5fffc97bdf-4h84r\" (UID: \"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:30:01.355832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355710    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6ec634f-81f4-4d4e-97ef-33092b38052a-ovnkube-config\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.356696 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.355965    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-certificates\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.356696 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.356029    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7427b4f7-fba8-4cde-ad90-1f46d5c92d87-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-hlvg8\" (UID: \"7427b4f7-fba8-4cde-ad90-1f46d5c92d87\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:30:01.361986 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.361958    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wkt8\" (UniqueName: \"kubernetes.io/projected/da310e19-8c58-4785-a88e-09edc8a63e2e-kube-api-access-4wkt8\") pod \"node-ca-lcxk5\" (UID: \"da310e19-8c58-4785-a88e-09edc8a63e2e\") " pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:30:01.362316 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.362289    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsl6n\" (UniqueName: \"kubernetes.io/projected/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-api-access-jsl6n\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:01.362739 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.362715    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-729fn\" (UniqueName: \"kubernetes.io/projected/e683558b-f046-46ae-99c4-4862e1247b03-kube-api-access-729fn\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.362809 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.362736    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdscq\" (UniqueName: \"kubernetes.io/projected/5db63b46-2839-43a5-9d7d-0cf80e8e1aa7-kube-api-access-fdscq\") pod \"migrator-66df5d496b-7k4fp\" (UID: \"5db63b46-2839-43a5-9d7d-0cf80e8e1aa7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7k4fp"
Oct 30 19:30:01.362809 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.362759    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfdmm\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-kube-api-access-zfdmm\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.362928 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.362885    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-bound-sa-token\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.363294 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.363263    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-bound-sa-token\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.363452 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.363435    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzkdg\" (UniqueName: \"kubernetes.io/projected/926e8fa4-1a22-48b1-832e-ea924ef61178-kube-api-access-rzkdg\") pod \"network-check-target-6wwsm\" (UID: \"926e8fa4-1a22-48b1-832e-ea924ef61178\") " pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:30:01.367266 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.367240    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6hxs\" (UniqueName: \"kubernetes.io/projected/b5f201ec-3c4d-4738-b721-ca4e445d63ad-kube-api-access-l6hxs\") pod \"node-resolver-2sdcp\" (UID: \"b5f201ec-3c4d-4738-b721-ca4e445d63ad\") " pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:30:01.367772 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.367747    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs72w\" (UniqueName: \"kubernetes.io/projected/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-kube-api-access-cs72w\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.367890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.367753    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2svq\" (UniqueName: \"kubernetes.io/projected/22a7299c-d539-47fa-87e2-626c5ca8bbbf-kube-api-access-g2svq\") pod \"periodic-gathering-pfjv8-8n5cz\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") " pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:30:01.368152 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.368134    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vxfk\" (UniqueName: \"kubernetes.io/projected/ac07aa06-d891-4c95-8500-fdef3723ff38-kube-api-access-5vxfk\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:30:01.368218 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.368191    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22dvx\" (UniqueName: \"kubernetes.io/projected/c6ec634f-81f4-4d4e-97ef-33092b38052a-kube-api-access-22dvx\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.368574 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.368550    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q76f\" (UniqueName: \"kubernetes.io/projected/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-kube-api-access-5q76f\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:01.368666 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.368581    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhbv6\" (UniqueName: \"kubernetes.io/projected/3615ac3b-837c-46ea-b575-4cdc1edc78a0-kube-api-access-jhbv6\") pod \"ingress-canary-vzpfb\" (UID: \"3615ac3b-837c-46ea-b575-4cdc1edc78a0\") " pod="openshift-ingress-canary/ingress-canary-vzpfb"
Oct 30 19:30:01.368666 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.368658    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcfz2\" (UniqueName: \"kubernetes.io/projected/52efc11e-b616-4c33-97d8-06fed7d75f82-kube-api-access-pcfz2\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.368854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.368836    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b8qf\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-kube-api-access-6b8qf\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.369436 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.369416    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zgxv\" (UniqueName: \"kubernetes.io/projected/bb3df8dc-d134-4cee-b95e-21e32f1df218-kube-api-access-8zgxv\") pod \"network-metrics-daemon-87b4p\" (UID: \"bb3df8dc-d134-4cee-b95e-21e32f1df218\") " pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:30:01.369532 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.369515    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkz6g\" (UniqueName: \"kubernetes.io/projected/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-kube-api-access-gkz6g\") pod \"iptables-alerter-f2s9d\" (UID: \"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5\") " pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:30:01.369839 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.369822    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjmx9\" (UniqueName: \"kubernetes.io/projected/4909e20c-2484-41bd-8f06-67f37dfd337b-kube-api-access-xjmx9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk\" (UID: \"4909e20c-2484-41bd-8f06-67f37dfd337b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk"
Oct 30 19:30:01.372416 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.372399    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72lkw\" (UniqueName: \"kubernetes.io/projected/ebaeaca4-4212-4bcc-b979-873495265bad-kube-api-access-72lkw\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.372814 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.372800    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99bgq\" (UniqueName: \"kubernetes.io/projected/98fe0d24-2073-47c4-af4a-171c54b7eb2d-kube-api-access-99bgq\") pod \"migrator-66df5d496b-2p5ts\" (UID: \"98fe0d24-2073-47c4-af4a-171c54b7eb2d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2p5ts"
Oct 30 19:30:01.373182 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.373166    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clprx\" (UniqueName: \"kubernetes.io/projected/ca3d3980-29e3-4afa-a9f8-302f62fb0ac4-kube-api-access-clprx\") pod \"service-ca-5fffc97bdf-4h84r\" (UID: \"ca3d3980-29e3-4afa-a9f8-302f62fb0ac4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4h84r"
Oct 30 19:30:01.456916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.456850    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-systemd-units\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.457057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.456922    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-run-ovn-kubernetes\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.457057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.456935    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-systemd-units\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.457057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.456942    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-cnibin\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.457057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.456990    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-cnibin\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.457057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.456997    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-cni-bin\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.457057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457017    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-conf-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.457057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457033    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-cni-netd\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.457057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457035    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-run-ovn-kubernetes\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.457057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457053    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457070    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-kubelet\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457086    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-registration-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457086    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-conf-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457110    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-system-cni-dir\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457107    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-cni-netd\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457102    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-cni-bin\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457140    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-system-cni-dir\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457141    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-log-socket\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457151    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457156    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-kubelet\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457170    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457168    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-registration-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457199    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-log-socket\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457251    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457264    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-sys\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457284    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-etc-selinux\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457303    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457319    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457333    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-ovn\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457342    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-sys\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457348    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysctl-conf\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457367    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-etc-selinux\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.457380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457377    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-ovn\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.457419    4823 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457446    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysctl-conf\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457443    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bwt58\" (UID: \"0c796373-b082-4069-b5e7-228588cc99da\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.457476    4823 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs podName:df3b6a17-4bbf-41dc-83e5-df5172b87b7f nodeName:}" failed. No retries permitted until 2026-10-30 19:30:01.957457096 +0000 UTC m=+1.850071416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs") pod "router-default-6cb4fdcd5-zsxtw" (UID: "df3b6a17-4bbf-41dc-83e5-df5172b87b7f") : secret "router-metrics-certs-default" not found
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457539    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-node-log\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457560    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-socket-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457575    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-device-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457605    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457630    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysconfig\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457638    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-node-log\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457643    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-socket-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457653    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-lib-modules\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457664    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-device-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457681    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5ff1aae-17b6-4563-949b-e8321be54e25-crio-socket\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457691    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysconfig\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457708    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.457714    4823 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457735    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-k8s-cni-cncf-io\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457735    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5ff1aae-17b6-4563-949b-e8321be54e25-crio-socket\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.457755    4823 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls podName:ac07aa06-d891-4c95-8500-fdef3723ff38 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:01.957740721 +0000 UTC m=+1.850355041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls") pod "dns-default-kl9jq" (UID: "ac07aa06-d891-4c95-8500-fdef3723ff38") : secret "dns-default-metrics-tls" not found
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457765    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-k8s-cni-cncf-io\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457791    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-kubernetes\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457791    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-lib-modules\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457817    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457833    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-kubernetes\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457892    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b5f201ec-3c4d-4738-b721-ca4e445d63ad-hosts-file\") pod \"node-resolver-2sdcp\" (UID: \"b5f201ec-3c4d-4738-b721-ca4e445d63ad\") " pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457915    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-sys-fs\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457943    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457954    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b5f201ec-3c4d-4738-b721-ca4e445d63ad-hosts-file\") pod \"node-resolver-2sdcp\" (UID: \"b5f201ec-3c4d-4738-b721-ca4e445d63ad\") " pod="openshift-dns/node-resolver-2sdcp"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457963    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-sys-fs\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457969    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-systemd\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.457913    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458000    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-run-systemd\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.458040    4823 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458046    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-run\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458077    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-var-lib-kubelet\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.458109    4823 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls podName:b5ff1aae-17b6-4563-949b-e8321be54e25 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:01.958092518 +0000 UTC m=+1.850706835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8nh5p" (UID: "b5ff1aae-17b6-4563-949b-e8321be54e25") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458108    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-run\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458132    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-var-lib-kubelet\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458142    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-cni-bin\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458169    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-run-netns\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458173    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-cni-bin\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458215    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-run-netns\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458216    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458234    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5ff1aae-17b6-4563-949b-e8321be54e25-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458326    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-netns\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458362    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-netns\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458367    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysctl-d\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458395    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-kubelet-config\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458422    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-etc-kubernetes\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458442    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-kubelet-config\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458456    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-sysctl-d\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458447    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-slash\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.458439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458480    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-etc-kubernetes\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458489    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-host-slash\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458494    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-cni-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458526    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-hostroot\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458533    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-cni-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458554    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458588    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-hostroot\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458595    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-var-lib-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458622    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-kubelet\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458651    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-systemd\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458655    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-kubelet\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458626    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-var-lib-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458675    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-dbus\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458701    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-etc-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458707    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-systemd\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458728    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-host-slash\") pod \"iptables-alerter-f2s9d\" (UID: \"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5\") " pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458754    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-cnibin\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458777    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-modprobe-d\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458790    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6ec634f-81f4-4d4e-97ef-33092b38052a-etc-openvswitch\") pod \"ovnkube-node-d4k5h\" (UID: \"c6ec634f-81f4-4d4e-97ef-33092b38052a\") " pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458800    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-host\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458823    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-system-cni-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458835    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52efc11e-b616-4c33-97d8-06fed7d75f82-cnibin\") pod \"multus-additional-cni-plugins-wg8f2\" (UID: \"52efc11e-b616-4c33-97d8-06fed7d75f82\") " pod="openshift-multus/multus-additional-cni-plugins-wg8f2"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458846    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-cni-multus\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458891    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs\") pod \"network-metrics-daemon-87b4p\" (UID: \"bb3df8dc-d134-4cee-b95e-21e32f1df218\") " pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458900    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5-host-slash\") pod \"iptables-alerter-f2s9d\" (UID: \"d4e7bd96-2bff-4a9c-9c6d-2a1e555975f5\") " pod="openshift-network-operator/iptables-alerter-f2s9d"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458918    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-socket-dir-parent\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458942    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-multus-certs\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458950    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-host\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458755    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e-dbus\") pod \"global-pull-secret-syncer-4zdqg\" (UID: \"bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e\") " pod="kube-system/global-pull-secret-syncer-4zdqg"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458965    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.458991    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da310e19-8c58-4785-a88e-09edc8a63e2e-host\") pod \"node-ca-lcxk5\" (UID: \"da310e19-8c58-4785-a88e-09edc8a63e2e\") " pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.459051    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e683558b-f046-46ae-99c4-4862e1247b03-etc-modprobe-d\") pod \"tuned-fcz7q\" (UID: \"e683558b-f046-46ae-99c4-4862e1247b03\") " pod="openshift-cluster-node-tuning-operator/tuned-fcz7q"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.459080    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da310e19-8c58-4785-a88e-09edc8a63e2e-host\") pod \"node-ca-lcxk5\" (UID: \"da310e19-8c58-4785-a88e-09edc8a63e2e\") " pod="openshift-image-registry/node-ca-lcxk5"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.459092    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-run-multus-certs\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.459130    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6f851f52-96a2-40cd-9ff0-ed8282ec1a0c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9n5d7\" (UID: \"6f851f52-96a2-40cd-9ff0-ed8282ec1a0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n5d7"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.459145    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-system-cni-dir\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.459181    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-multus-socket-dir-parent\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.459191    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ebaeaca4-4212-4bcc-b979-873495265bad-host-var-lib-cni-multus\") pod \"multus-k6rd4\" (UID: \"ebaeaca4-4212-4bcc-b979-873495265bad\") " pod="openshift-multus/multus-k6rd4"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.459448    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-service-ca-bundle\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.460510    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls\") pod \"image-registry-5fdc8795f9-cfh7p\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") " pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.460557    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c796373-b082-4069-b5e7-228588cc99da-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bwt58\" (UID: \"0c796373-b082-4069-b5e7-228588cc99da\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:30:01.460902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.460574    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls\") pod \"image-registry-7ff684fc5d-nksx9\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") " pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.462416 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.461998    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3df8dc-d134-4cee-b95e-21e32f1df218-metrics-certs\") pod \"network-metrics-daemon-87b4p\" (UID: \"bb3df8dc-d134-4cee-b95e-21e32f1df218\") " pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:30:01.533684 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.533601    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"
Oct 30 19:30:01.535782 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.535760    4823 scope.go:122] "RemoveContainer" containerID="7faa03714fb97f461d9afe0d24129275695c5969490602c53524e726085fcc81"
Oct 30 19:30:01.537375 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.537307    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:01.538472 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.538454    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58"
Oct 30 19:30:01.549569 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.549538    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87b4p"
Oct 30 19:30:01.555200 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.553818    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:01.848133 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.847883    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8"]
Oct 30 19:30:01.852545 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:01.852516    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7427b4f7_fba8_4cde_ad90_1f46d5c92d87.slice/crio-0f4633f4a1ba7be0db6f02fe14096bd215895efa690544fb4368521863948797 WatchSource:0}: Error finding container 0f4633f4a1ba7be0db6f02fe14096bd215895efa690544fb4368521863948797: Status 404 returned error can't find the container with id 0f4633f4a1ba7be0db6f02fe14096bd215895efa690544fb4368521863948797
Oct 30 19:30:01.857175 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.857144    4823 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:30:01.862804 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.862780    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:30:01.862948 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.862930    4823 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:30:01.863000 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.862991    4823 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls podName:f387d9b5-577c-470c-a25e-da28234f464a nodeName:}" failed. No retries permitted until 2026-10-30 19:30:02.862977675 +0000 UTC m=+2.755591999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xdndw" (UID: "f387d9b5-577c-470c-a25e-da28234f464a") : secret "samples-operator-tls" not found
Oct 30 19:30:01.895727 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.895689    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7ff684fc5d-nksx9"]
Oct 30 19:30:01.899620 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:01.899590    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0ec109_980a_4203_b45b_25ebd01ae8ed.slice/crio-795d88aef4e175b414633a2097adc65c857beb6defcf3f81e840dd0715b93cfa WatchSource:0}: Error finding container 795d88aef4e175b414633a2097adc65c857beb6defcf3f81e840dd0715b93cfa: Status 404 returned error can't find the container with id 795d88aef4e175b414633a2097adc65c857beb6defcf3f81e840dd0715b93cfa
Oct 30 19:30:01.943765 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.943720    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fdc8795f9-cfh7p"]
Oct 30 19:30:01.947782 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:01.947747    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153c1fd1_90a2_4063_b066_6503d796d6d8.slice/crio-e51682e4319eef873d84ebeb6b159a14ea2a3400f45a680a73a7e1e11eb00605 WatchSource:0}: Error finding container e51682e4319eef873d84ebeb6b159a14ea2a3400f45a680a73a7e1e11eb00605: Status 404 returned error can't find the container with id e51682e4319eef873d84ebeb6b159a14ea2a3400f45a680a73a7e1e11eb00605
Oct 30 19:30:01.949815 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.949733    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-bwt58"]
Oct 30 19:30:01.952500 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:01.952476    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c796373_b082_4069_b5e7_228588cc99da.slice/crio-4cd7eac02b1fda8c5197b1cc0cdd80a31b68649444cd86f3b565a8874cf4911c WatchSource:0}: Error finding container 4cd7eac02b1fda8c5197b1cc0cdd80a31b68649444cd86f3b565a8874cf4911c: Status 404 returned error can't find the container with id 4cd7eac02b1fda8c5197b1cc0cdd80a31b68649444cd86f3b565a8874cf4911c
Oct 30 19:30:01.961951 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.961926    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-87b4p"]
Oct 30 19:30:01.963504 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.963481    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:01.963566 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.963548    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:30:01.963658 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:01.963600    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:01.963718 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.963651    4823 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:30:01.963772 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.963718    4823 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs podName:df3b6a17-4bbf-41dc-83e5-df5172b87b7f nodeName:}" failed. No retries permitted until 2026-10-30 19:30:02.963698335 +0000 UTC m=+2.856312640 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs") pod "router-default-6cb4fdcd5-zsxtw" (UID: "df3b6a17-4bbf-41dc-83e5-df5172b87b7f") : secret "router-metrics-certs-default" not found
Oct 30 19:30:01.963772 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.963757    4823 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:30:01.963887 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.963811    4823 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls podName:b5ff1aae-17b6-4563-949b-e8321be54e25 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:02.963793946 +0000 UTC m=+2.856408263 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8nh5p" (UID: "b5ff1aae-17b6-4563-949b-e8321be54e25") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:30:01.963942 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.963904    4823 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:30:01.963989 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:01.963949    4823 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls podName:ac07aa06-d891-4c95-8500-fdef3723ff38 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:02.963936686 +0000 UTC m=+2.856550999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls") pod "dns-default-kl9jq" (UID: "ac07aa06-d891-4c95-8500-fdef3723ff38") : secret "dns-default-metrics-tls" not found
Oct 30 19:30:01.965760 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:01.965737    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3df8dc_d134_4cee_b95e_21e32f1df218.slice/crio-50845e125c23b2f65759eaaf7cdf4ce28bdad68e395191eaf00e0373ee1377ef WatchSource:0}: Error finding container 50845e125c23b2f65759eaaf7cdf4ce28bdad68e395191eaf00e0373ee1377ef: Status 404 returned error can't find the container with id 50845e125c23b2f65759eaaf7cdf4ce28bdad68e395191eaf00e0373ee1377ef
Oct 30 19:30:02.349509 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:02.349469    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87b4p" event={"ID":"bb3df8dc-d134-4cee-b95e-21e32f1df218","Type":"ContainerStarted","Data":"50845e125c23b2f65759eaaf7cdf4ce28bdad68e395191eaf00e0373ee1377ef"}
Oct 30 19:30:02.351269 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:02.351238    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8" event={"ID":"7427b4f7-fba8-4cde-ad90-1f46d5c92d87","Type":"ContainerStarted","Data":"0f4633f4a1ba7be0db6f02fe14096bd215895efa690544fb4368521863948797"}
Oct 30 19:30:02.353940 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:02.353917    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 19:30:02.354197 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:02.354176    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx" event={"ID":"392e4212-393d-4aaa-904b-f72f1e287c34","Type":"ContainerStarted","Data":"726eb6a9575a1f1d237added1f065011ade55b230ac5f62c7f8d5b712e9c4aeb"}
Oct 30 19:30:02.370393 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:02.370160    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9" event={"ID":"9e0ec109-980a-4203-b45b-25ebd01ae8ed","Type":"ContainerStarted","Data":"64496385aad5574c7eeede13b9922e403af1dd92a447852b95956ae876c8031e"}
Oct 30 19:30:02.370393 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:02.370306    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9" event={"ID":"9e0ec109-980a-4203-b45b-25ebd01ae8ed","Type":"ContainerStarted","Data":"795d88aef4e175b414633a2097adc65c857beb6defcf3f81e840dd0715b93cfa"}
Oct 30 19:30:02.373788 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:02.373754    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p" event={"ID":"153c1fd1-90a2-4063-b066-6503d796d6d8","Type":"ContainerStarted","Data":"af0167644ab6253695161cd09fbe1c2d98f0a9b26ee8b2d761a568e5e97478ab"}
Oct 30 19:30:02.373984 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:02.373835    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p" event={"ID":"153c1fd1-90a2-4063-b066-6503d796d6d8","Type":"ContainerStarted","Data":"e51682e4319eef873d84ebeb6b159a14ea2a3400f45a680a73a7e1e11eb00605"}
Oct 30 19:30:02.375266 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:02.375232    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58" event={"ID":"0c796373-b082-4069-b5e7-228588cc99da","Type":"ContainerStarted","Data":"4cd7eac02b1fda8c5197b1cc0cdd80a31b68649444cd86f3b565a8874cf4911c"}
Oct 30 19:30:02.622276 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:02.622241    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:30:02.636934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:02.636896    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-dp8wx"
Oct 30 19:30:02.877744 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:02.877647    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:30:02.894652 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:02.894620    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f387d9b5-577c-470c-a25e-da28234f464a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xdndw\" (UID: \"f387d9b5-577c-470c-a25e-da28234f464a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:30:02.979274 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:02.979226    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:02.979439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:02.979288    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:30:02.979439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:02.979335    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:02.979545 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:02.979496    4823 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:30:02.979586 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:02.979563    4823 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls podName:b5ff1aae-17b6-4563-949b-e8321be54e25 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:04.979542783 +0000 UTC m=+4.872157101 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8nh5p" (UID: "b5ff1aae-17b6-4563-949b-e8321be54e25") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:30:02.980001 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:02.979984    4823 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:30:02.980051 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:02.980037    4823 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs podName:df3b6a17-4bbf-41dc-83e5-df5172b87b7f nodeName:}" failed. No retries permitted until 2026-10-30 19:30:04.980021744 +0000 UTC m=+4.872636050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs") pod "router-default-6cb4fdcd5-zsxtw" (UID: "df3b6a17-4bbf-41dc-83e5-df5172b87b7f") : secret "router-metrics-certs-default" not found
Oct 30 19:30:02.980110 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:02.980097    4823 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:30:02.980147 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:02.980137    4823 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls podName:ac07aa06-d891-4c95-8500-fdef3723ff38 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:04.98012467 +0000 UTC m=+4.872738991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls") pod "dns-default-kl9jq" (UID: "ac07aa06-d891-4c95-8500-fdef3723ff38") : secret "dns-default-metrics-tls" not found
Oct 30 19:30:03.034574 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:03.034538    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"
Oct 30 19:30:03.339248 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:03.339218    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:03.367929 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:03.367826    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw"]
Oct 30 19:30:03.381193 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:03.381129    4823 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:30:03.381193 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:03.381129    4823 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:30:04.389515 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:04.389246    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-bwt58" event={"ID":"0c796373-b082-4069-b5e7-228588cc99da","Type":"ContainerStarted","Data":"a53862e2a36fbcf869fec9e9956ff6bb96b2096d94b348356e7b28c505c86521"}
Oct 30 19:30:04.391720 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:04.391686    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87b4p" event={"ID":"bb3df8dc-d134-4cee-b95e-21e32f1df218","Type":"ContainerStarted","Data":"42425eb5c1dc73ff1f4cb662003fed8bf25519fb9cf6fa974ed2f6bd9f256432"}
Oct 30 19:30:04.391849 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:04.391749    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87b4p" event={"ID":"bb3df8dc-d134-4cee-b95e-21e32f1df218","Type":"ContainerStarted","Data":"8f8fc75af3297e90dd6af2460948d8bb05c9b3662d06264dd20125214bbdf779"}
Oct 30 19:30:04.393823 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:04.393769    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-hlvg8" event={"ID":"7427b4f7-fba8-4cde-ad90-1f46d5c92d87","Type":"ContainerStarted","Data":"c49d7669d4ad4d5350cc974792af7e68e4d05f60c472cacfe4897d252ec51a0b"}
Oct 30 19:30:04.395488 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:04.395407    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw" event={"ID":"f387d9b5-577c-470c-a25e-da28234f464a","Type":"ContainerStarted","Data":"deb8cb935a1f5630c1d0a595af82cdb0a3ba96ed2b0bc8de19adebb7386f846a"}
Oct 30 19:30:04.788495 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:04.788416    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:04.788691 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:04.788674    4823 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:30:05.000807 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:05.000762    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:05.001017 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:05.000824    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:30:05.001017 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:05.000880    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:05.001131 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:05.001018    4823 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:30:05.001131 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:05.001083    4823 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls podName:b5ff1aae-17b6-4563-949b-e8321be54e25 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:09.001064158 +0000 UTC m=+8.893678463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8nh5p" (UID: "b5ff1aae-17b6-4563-949b-e8321be54e25") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:30:05.004881 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:05.004837    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac07aa06-d891-4c95-8500-fdef3723ff38-metrics-tls\") pod \"dns-default-kl9jq\" (UID: \"ac07aa06-d891-4c95-8500-fdef3723ff38\") " pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:30:05.005025 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:05.004888    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df3b6a17-4bbf-41dc-83e5-df5172b87b7f-metrics-certs\") pod \"router-default-6cb4fdcd5-zsxtw\" (UID: \"df3b6a17-4bbf-41dc-83e5-df5172b87b7f\") " pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:05.136138 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:05.136110    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:05.154138 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:05.154105    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:30:05.400988 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:05.400906    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw" event={"ID":"f387d9b5-577c-470c-a25e-da28234f464a","Type":"ContainerStarted","Data":"08e105a4fb5d1046588e4f572a38f4422a8d3cde39d8a0d16f6351b743d45f5a"}
Oct 30 19:30:05.400988 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:05.400968    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xdndw" event={"ID":"f387d9b5-577c-470c-a25e-da28234f464a","Type":"ContainerStarted","Data":"1fb6a349a92479c5c21a73f9d81fc1938f3a62eb5d449ae3912aa691130d5598"}
Oct 30 19:30:05.402515 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:05.402491    4823 generic.go:356] "Generic (PLEG): container finished" podID="4909e20c-2484-41bd-8f06-67f37dfd337b" containerID="c65d83d460dfa1b48b3ab32eabd819f4955e7df61c3c6563239873c559b80285" exitCode=0
Oct 30 19:30:05.402615 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:05.402588    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk" event={"ID":"4909e20c-2484-41bd-8f06-67f37dfd337b","Type":"ContainerDied","Data":"c65d83d460dfa1b48b3ab32eabd819f4955e7df61c3c6563239873c559b80285"}
Oct 30 19:30:05.403073 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:05.403055    4823 scope.go:122] "RemoveContainer" containerID="c65d83d460dfa1b48b3ab32eabd819f4955e7df61c3c6563239873c559b80285"
Oct 30 19:30:05.433230 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:05.433200    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6cb4fdcd5-zsxtw"]
Oct 30 19:30:05.439064 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:05.439036    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3b6a17_4bbf_41dc_83e5_df5172b87b7f.slice/crio-6945cc9f172d416aee90f9e56e3f6c350717287a168a93b6b457750d4aad91d5 WatchSource:0}: Error finding container 6945cc9f172d416aee90f9e56e3f6c350717287a168a93b6b457750d4aad91d5: Status 404 returned error can't find the container with id 6945cc9f172d416aee90f9e56e3f6c350717287a168a93b6b457750d4aad91d5
Oct 30 19:30:05.451209 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:05.451156    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kl9jq"]
Oct 30 19:30:05.455204 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:05.455167    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac07aa06_d891_4c95_8500_fdef3723ff38.slice/crio-089f9c4505ccb54d8c1c1e680bf9548a1556997b40f63bbc528206b05021e1ca WatchSource:0}: Error finding container 089f9c4505ccb54d8c1c1e680bf9548a1556997b40f63bbc528206b05021e1ca: Status 404 returned error can't find the container with id 089f9c4505ccb54d8c1c1e680bf9548a1556997b40f63bbc528206b05021e1ca
Oct 30 19:30:06.408601 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:06.408554    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kl9jq" event={"ID":"ac07aa06-d891-4c95-8500-fdef3723ff38","Type":"ContainerStarted","Data":"089f9c4505ccb54d8c1c1e680bf9548a1556997b40f63bbc528206b05021e1ca"}
Oct 30 19:30:06.410592 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:06.410556    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw" event={"ID":"df3b6a17-4bbf-41dc-83e5-df5172b87b7f","Type":"ContainerStarted","Data":"fd7301f414f00650d780d51db52cf9169f9df0c0951cb9d25fd15c74cfb9d3af"}
Oct 30 19:30:06.410722 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:06.410617    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw" event={"ID":"df3b6a17-4bbf-41dc-83e5-df5172b87b7f","Type":"ContainerStarted","Data":"6945cc9f172d416aee90f9e56e3f6c350717287a168a93b6b457750d4aad91d5"}
Oct 30 19:30:06.412930 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:06.412901    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-d4lhk" event={"ID":"4909e20c-2484-41bd-8f06-67f37dfd337b","Type":"ContainerStarted","Data":"6aa4531b207d02e57ea90786488e46b0f87ded72a1e29601a1ec249c07d120f2"}
Oct 30 19:30:07.137259 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:07.137220    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:07.140639 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:07.140611    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:07.418810 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:07.418717    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kl9jq" event={"ID":"ac07aa06-d891-4c95-8500-fdef3723ff38","Type":"ContainerStarted","Data":"372486339d1128c772a73629de0ea8cdd003e3d475b7dccd932e71b4f1ac893a"}
Oct 30 19:30:07.418810 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:07.418779    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kl9jq" event={"ID":"ac07aa06-d891-4c95-8500-fdef3723ff38","Type":"ContainerStarted","Data":"6fed4c1ee52baa51d0ca1f66011036f007c6eb0f8275c070d5dfdb43ee897652"}
Oct 30 19:30:07.419286 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:07.418819    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:30:07.950568 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:07.950523    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:07.951951 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:07.951926    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6cb4fdcd5-zsxtw"
Oct 30 19:30:08.383847 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:08.383813    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:30:09.035085 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:09.035044    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:09.037393 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:09.037357    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5ff1aae-17b6-4563-949b-e8321be54e25-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nh5p\" (UID: \"b5ff1aae-17b6-4563-949b-e8321be54e25\") " pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:09.059400 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:09.059363    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8nh5p"
Oct 30 19:30:09.337854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:09.337773    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8nh5p"]
Oct 30 19:30:09.341854 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:09.341825    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ff1aae_17b6_4563_949b_e8321be54e25.slice/crio-1591d60a146551682a77fd615fff92dde9ac7c9d0353a8b712c07ca52bb3dfe2 WatchSource:0}: Error finding container 1591d60a146551682a77fd615fff92dde9ac7c9d0353a8b712c07ca52bb3dfe2: Status 404 returned error can't find the container with id 1591d60a146551682a77fd615fff92dde9ac7c9d0353a8b712c07ca52bb3dfe2
Oct 30 19:30:09.429290 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:09.429249    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8nh5p" event={"ID":"b5ff1aae-17b6-4563-949b-e8321be54e25","Type":"ContainerStarted","Data":"4e691f7968564138af321486d8c962f49f767854faff7fb12f0821cf1ce8cce7"}
Oct 30 19:30:09.429476 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:09.429327    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8nh5p" event={"ID":"b5ff1aae-17b6-4563-949b-e8321be54e25","Type":"ContainerStarted","Data":"1591d60a146551682a77fd615fff92dde9ac7c9d0353a8b712c07ca52bb3dfe2"}
Oct 30 19:30:10.435909 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:10.435855    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8nh5p" event={"ID":"b5ff1aae-17b6-4563-949b-e8321be54e25","Type":"ContainerStarted","Data":"b628cfad2655c8d979ffe5339fbe402509c874b6dbf4ab3207328b1bb7bac29d"}
Oct 30 19:30:10.511749 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:10.511708    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:11.214142 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:11.214089    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:12.447395 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:12.447360    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8nh5p" event={"ID":"b5ff1aae-17b6-4563-949b-e8321be54e25","Type":"ContainerStarted","Data":"52213965a2192b6eef3c868a3213f49374bf5b7f69c3ac35bc477a8ae110e1d6"}
Oct 30 19:30:15.390643 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:15.390587    4823 patch_prober.go:29] interesting pod/image-registry-5fdc8795f9-cfh7p container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 19:30:15.391111 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:15.390660    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p" podUID="153c1fd1-90a2-4063-b066-6503d796d6d8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:30:16.308763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.308729    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-xqs7x"]
Oct 30 19:30:16.312243 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.312222    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-xqs7x"
Oct 30 19:30:16.319495 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.319455    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5s2v6"]
Oct 30 19:30:16.322999 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.322976    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5s2v6"
Oct 30 19:30:16.333474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.333427    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 19:30:16.333474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.333477    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 19:30:16.338698 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.338658    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ggfp6\""
Oct 30 19:30:16.342707 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.342679    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 19:30:16.342873 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.342787    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5s2v6"]
Oct 30 19:30:16.343088 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.343068    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-pp89r\""
Oct 30 19:30:16.354771 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.354741    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7ff684fc5d-nksx9"]
Oct 30 19:30:16.356600 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.356577    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-xqs7x"]
Oct 30 19:30:16.360676 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.360627    4823 patch_prober.go:29] interesting pod/image-registry-7ff684fc5d-nksx9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 19:30:16.360816 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.360683    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9" podUID="9e0ec109-980a-4203-b45b-25ebd01ae8ed" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:30:16.399582 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.399544    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6945cea2-f314-48c5-b638-1b63a269a3f6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5s2v6\" (UID: \"6945cea2-f314-48c5-b638-1b63a269a3f6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5s2v6"
Oct 30 19:30:16.400088 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.399586    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c48252d-5c0b-42fd-9e6c-c93d3e593379-tmp\") pod \"downloads-6dd789c8b7-xqs7x\" (UID: \"2c48252d-5c0b-42fd-9e6c-c93d3e593379\") " pod="openshift-console/downloads-6dd789c8b7-xqs7x"
Oct 30 19:30:16.400088 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.399937    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fl9m\" (UniqueName: \"kubernetes.io/projected/2c48252d-5c0b-42fd-9e6c-c93d3e593379-kube-api-access-8fl9m\") pod \"downloads-6dd789c8b7-xqs7x\" (UID: \"2c48252d-5c0b-42fd-9e6c-c93d3e593379\") " pod="openshift-console/downloads-6dd789c8b7-xqs7x"
Oct 30 19:30:16.418088 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.418058    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cf748dbd6-ljbz6"]
Oct 30 19:30:16.421317 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.421298    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.439400 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.439362    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cf748dbd6-ljbz6"]
Oct 30 19:30:16.501175 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.501139    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fl9m\" (UniqueName: \"kubernetes.io/projected/2c48252d-5c0b-42fd-9e6c-c93d3e593379-kube-api-access-8fl9m\") pod \"downloads-6dd789c8b7-xqs7x\" (UID: \"2c48252d-5c0b-42fd-9e6c-c93d3e593379\") " pod="openshift-console/downloads-6dd789c8b7-xqs7x"
Oct 30 19:30:16.501175 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.501177    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/899ea1d2-97ca-46c0-988a-886f70a01bed-registry-tls\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.501406 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.501197    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/899ea1d2-97ca-46c0-988a-886f70a01bed-registry-certificates\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.501406 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.501222    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxj8p\" (UniqueName: \"kubernetes.io/projected/899ea1d2-97ca-46c0-988a-886f70a01bed-kube-api-access-bxj8p\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.501406 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.501322    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/899ea1d2-97ca-46c0-988a-886f70a01bed-bound-sa-token\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.501406 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.501352    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6945cea2-f314-48c5-b638-1b63a269a3f6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5s2v6\" (UID: \"6945cea2-f314-48c5-b638-1b63a269a3f6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5s2v6"
Oct 30 19:30:16.501406 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.501375    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c48252d-5c0b-42fd-9e6c-c93d3e593379-tmp\") pod \"downloads-6dd789c8b7-xqs7x\" (UID: \"2c48252d-5c0b-42fd-9e6c-c93d3e593379\") " pod="openshift-console/downloads-6dd789c8b7-xqs7x"
Oct 30 19:30:16.501406 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.501392    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/899ea1d2-97ca-46c0-988a-886f70a01bed-ca-trust-extracted\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.501616 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.501446    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/899ea1d2-97ca-46c0-988a-886f70a01bed-image-registry-private-configuration\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.501616 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.501477    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/899ea1d2-97ca-46c0-988a-886f70a01bed-installation-pull-secrets\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.501709 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.501689    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/899ea1d2-97ca-46c0-988a-886f70a01bed-trusted-ca\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.501943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.501900    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c48252d-5c0b-42fd-9e6c-c93d3e593379-tmp\") pod \"downloads-6dd789c8b7-xqs7x\" (UID: \"2c48252d-5c0b-42fd-9e6c-c93d3e593379\") " pod="openshift-console/downloads-6dd789c8b7-xqs7x"
Oct 30 19:30:16.503979 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.503954    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6945cea2-f314-48c5-b638-1b63a269a3f6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5s2v6\" (UID: \"6945cea2-f314-48c5-b638-1b63a269a3f6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5s2v6"
Oct 30 19:30:16.513685 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.513658    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fl9m\" (UniqueName: \"kubernetes.io/projected/2c48252d-5c0b-42fd-9e6c-c93d3e593379-kube-api-access-8fl9m\") pod \"downloads-6dd789c8b7-xqs7x\" (UID: \"2c48252d-5c0b-42fd-9e6c-c93d3e593379\") " pod="openshift-console/downloads-6dd789c8b7-xqs7x"
Oct 30 19:30:16.603020 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.602930    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/899ea1d2-97ca-46c0-988a-886f70a01bed-registry-tls\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.603020 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.602974    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/899ea1d2-97ca-46c0-988a-886f70a01bed-registry-certificates\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.603020 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.603000    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxj8p\" (UniqueName: \"kubernetes.io/projected/899ea1d2-97ca-46c0-988a-886f70a01bed-kube-api-access-bxj8p\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.603252 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.603133    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/899ea1d2-97ca-46c0-988a-886f70a01bed-bound-sa-token\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.603252 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.603214    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/899ea1d2-97ca-46c0-988a-886f70a01bed-ca-trust-extracted\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.603323 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.603255    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/899ea1d2-97ca-46c0-988a-886f70a01bed-image-registry-private-configuration\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.603323 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.603285    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/899ea1d2-97ca-46c0-988a-886f70a01bed-installation-pull-secrets\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.603421 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.603365    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/899ea1d2-97ca-46c0-988a-886f70a01bed-trusted-ca\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.603583 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.603552    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/899ea1d2-97ca-46c0-988a-886f70a01bed-ca-trust-extracted\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.603975 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.603948    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/899ea1d2-97ca-46c0-988a-886f70a01bed-registry-certificates\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.604178 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.604160    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/899ea1d2-97ca-46c0-988a-886f70a01bed-trusted-ca\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.605666 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.605635    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/899ea1d2-97ca-46c0-988a-886f70a01bed-installation-pull-secrets\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.605666 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.605664    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/899ea1d2-97ca-46c0-988a-886f70a01bed-registry-tls\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.605821 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.605678    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/899ea1d2-97ca-46c0-988a-886f70a01bed-image-registry-private-configuration\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.617027 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.617000    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/899ea1d2-97ca-46c0-988a-886f70a01bed-bound-sa-token\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.617144 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.617125    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxj8p\" (UniqueName: \"kubernetes.io/projected/899ea1d2-97ca-46c0-988a-886f70a01bed-kube-api-access-bxj8p\") pod \"image-registry-7cf748dbd6-ljbz6\" (UID: \"899ea1d2-97ca-46c0-988a-886f70a01bed\") " pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.626770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.626743    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-xqs7x"
Oct 30 19:30:16.636193 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.636167    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5s2v6"
Oct 30 19:30:16.734047 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.734006    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:16.932822 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.932789    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5s2v6"]
Oct 30 19:30:16.934688 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:16.934659    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-xqs7x"]
Oct 30 19:30:16.937146 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:16.937111    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6945cea2_f314_48c5_b638_1b63a269a3f6.slice/crio-29dfdb3345d9944c832edd529173782fb75b2dd6903df2a309b1ca4706ef5509 WatchSource:0}: Error finding container 29dfdb3345d9944c832edd529173782fb75b2dd6903df2a309b1ca4706ef5509: Status 404 returned error can't find the container with id 29dfdb3345d9944c832edd529173782fb75b2dd6903df2a309b1ca4706ef5509
Oct 30 19:30:16.937968 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:16.937942    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c48252d_5c0b_42fd_9e6c_c93d3e593379.slice/crio-4cc6f8a4ad2680caa223215a331a247ede6892c7dde8f9d5ff0e33842bb2af35 WatchSource:0}: Error finding container 4cc6f8a4ad2680caa223215a331a247ede6892c7dde8f9d5ff0e33842bb2af35: Status 404 returned error can't find the container with id 4cc6f8a4ad2680caa223215a331a247ede6892c7dde8f9d5ff0e33842bb2af35
Oct 30 19:30:17.019072 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:17.019040    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cf748dbd6-ljbz6"]
Oct 30 19:30:17.022813 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:17.022785    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899ea1d2_97ca_46c0_988a_886f70a01bed.slice/crio-af025b117d0758c18e5f153db80161762ea04a304ea9d76ed7c94e2497bdd321 WatchSource:0}: Error finding container af025b117d0758c18e5f153db80161762ea04a304ea9d76ed7c94e2497bdd321: Status 404 returned error can't find the container with id af025b117d0758c18e5f153db80161762ea04a304ea9d76ed7c94e2497bdd321
Oct 30 19:30:17.425564 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:17.425533    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kl9jq"
Oct 30 19:30:17.469751 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:17.469714    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-xqs7x" event={"ID":"2c48252d-5c0b-42fd-9e6c-c93d3e593379","Type":"ContainerStarted","Data":"4cc6f8a4ad2680caa223215a331a247ede6892c7dde8f9d5ff0e33842bb2af35"}
Oct 30 19:30:17.472124 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:17.472073    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6" event={"ID":"899ea1d2-97ca-46c0-988a-886f70a01bed","Type":"ContainerStarted","Data":"5d186504c1ab3c1cf04ea07f3503987e604c21fd8a28f86ea2e3d87a6fb971e0"}
Oct 30 19:30:17.472277 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:17.472138    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6" event={"ID":"899ea1d2-97ca-46c0-988a-886f70a01bed","Type":"ContainerStarted","Data":"af025b117d0758c18e5f153db80161762ea04a304ea9d76ed7c94e2497bdd321"}
Oct 30 19:30:17.472361 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:17.472338    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:17.473797 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:17.473772    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5s2v6" event={"ID":"6945cea2-f314-48c5-b638-1b63a269a3f6","Type":"ContainerStarted","Data":"29dfdb3345d9944c832edd529173782fb75b2dd6903df2a309b1ca4706ef5509"}
Oct 30 19:30:17.501956 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:17.501902    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6" podStartSLOduration=1.501883684 podStartE2EDuration="1.501883684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:17.500139353 +0000 UTC m=+17.392753677" watchObservedRunningTime="2026-10-30 19:30:17.501883684 +0000 UTC m=+17.394498011"
Oct 30 19:30:18.482040 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:18.481941    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5s2v6" event={"ID":"6945cea2-f314-48c5-b638-1b63a269a3f6","Type":"ContainerStarted","Data":"972a965497dd9879c36c186da8d14d335b6881e5ad43a59b855714b8d8ca6b31"}
Oct 30 19:30:18.482040 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:18.482018    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5s2v6"
Oct 30 19:30:18.488395 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:18.488361    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5s2v6"
Oct 30 19:30:18.536435 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:18.536366    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5s2v6" podStartSLOduration=1.247466494 podStartE2EDuration="2.536344926s" totalImagesPullingTime="1.288878432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:18.534623853 +0000 UTC m=+18.427238173" watchObservedRunningTime="2026-10-30 19:30:18.536344926 +0000 UTC m=+18.428959249"
Oct 30 19:30:20.871415 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:20.871379    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d4k5h"
Oct 30 19:30:21.219825 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:21.219731    4823 patch_prober.go:29] interesting pod/image-registry-5fdc8795f9-cfh7p container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 19:30:21.219825 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:21.219797    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p" podUID="153c1fd1-90a2-4063-b066-6503d796d6d8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:30:24.209825 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.209782    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j4tbd"]
Oct 30 19:30:24.242197 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.242167    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.252232 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.252207    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ch4qh\""
Oct 30 19:30:24.252232 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.252207    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:30:24.252420 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.252244    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:30:24.254506 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.254476    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:30:24.254626 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.254598    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:30:24.377626 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.377586    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-node-exporter-accelerators-collector-config\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.377816 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.377644    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-sys\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.377816 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.377750    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-metrics-client-ca\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.377929 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.377857    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2gw2\" (UniqueName: \"kubernetes.io/projected/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-kube-api-access-p2gw2\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.377929 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.377915    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-node-exporter-textfile\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.378029 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.378014    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.378066 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.378045    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-node-exporter-tls\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.378119 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.378104    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-node-exporter-wtmp\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.378173 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.378129    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-root\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.478934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.478758    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-node-exporter-accelerators-collector-config\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.478934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.478840    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-sys\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.478934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.478901    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-metrics-client-ca\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.478934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.478931    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2gw2\" (UniqueName: \"kubernetes.io/projected/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-kube-api-access-p2gw2\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.479330 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.478967    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-node-exporter-textfile\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.479330 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.478957    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-sys\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.479330 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.479008    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.479330 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.479056    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-node-exporter-tls\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.479330 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.479085    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-node-exporter-wtmp\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.479330 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.479113    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-root\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.479330 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.479191    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-root\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.479650 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.479365    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-node-exporter-textfile\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.479650 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.479370    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-node-exporter-wtmp\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.479650 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.479519    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-node-exporter-accelerators-collector-config\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.479756 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.479658    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-metrics-client-ca\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.481776 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.481748    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-node-exporter-tls\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.481921 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.481823    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.493389 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.493362    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2gw2\" (UniqueName: \"kubernetes.io/projected/fe1b3ffd-27d4-46fe-80e8-5920b5b62dac-kube-api-access-p2gw2\") pod \"node-exporter-j4tbd\" (UID: \"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac\") " pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.612422 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:24.612380    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j4tbd"
Oct 30 19:30:24.651469 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:30:24.651438    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1b3ffd_27d4_46fe_80e8_5920b5b62dac.slice/crio-d122f3079c9a2f8637e209124a0844f27186befda8da5a9627853a34969fb5f0 WatchSource:0}: Error finding container d122f3079c9a2f8637e209124a0844f27186befda8da5a9627853a34969fb5f0: Status 404 returned error can't find the container with id d122f3079c9a2f8637e209124a0844f27186befda8da5a9627853a34969fb5f0
Oct 30 19:30:25.522921 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:25.521339    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j4tbd" event={"ID":"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac","Type":"ContainerStarted","Data":"d122f3079c9a2f8637e209124a0844f27186befda8da5a9627853a34969fb5f0"}
Oct 30 19:30:26.360944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:26.360910    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:26.528393 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:26.528311    4823 generic.go:356] "Generic (PLEG): container finished" podID="fe1b3ffd-27d4-46fe-80e8-5920b5b62dac" containerID="becadfcff2e0e970b3824ee49d2061c52b8f05642b7bebcbe930f1e6d581054b" exitCode=0
Oct 30 19:30:26.528958 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:26.528392    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j4tbd" event={"ID":"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac","Type":"ContainerDied","Data":"becadfcff2e0e970b3824ee49d2061c52b8f05642b7bebcbe930f1e6d581054b"}
Oct 30 19:30:27.535939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:27.535895    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j4tbd" event={"ID":"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac","Type":"ContainerStarted","Data":"e750b9542fb7b54e564a4a22b6b2acda3a4f2805e13b085beb3f71df01f37dc0"}
Oct 30 19:30:27.536408 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:27.535967    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j4tbd" event={"ID":"fe1b3ffd-27d4-46fe-80e8-5920b5b62dac","Type":"ContainerStarted","Data":"d32523226b2217af8aaa2b3ab6ed081f10465aa1121e58a162256c439b2ae26b"}
Oct 30 19:30:27.556324 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:27.556264    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j4tbd" podStartSLOduration=1.943201026 podStartE2EDuration="3.556248301s" totalImagesPullingTime="1.526086207s" totalInitContainerRuntime="86.961068ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:27.555478371 +0000 UTC m=+27.448092697" watchObservedRunningTime="2026-10-30 19:30:27.556248301 +0000 UTC m=+27.448862624"
Oct 30 19:30:28.387287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:28.387253    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6wwsm"
Oct 30 19:30:30.710323 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:30.710286    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ncbz9"
Oct 30 19:30:31.219794 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:31.219763    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:30:32.016618 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:30:32.016578    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:30:34.571680 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:34.571638    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-xqs7x" event={"ID":"2c48252d-5c0b-42fd-9e6c-c93d3e593379","Type":"ContainerStarted","Data":"41ffca77c2ac22c5863143cce9e60b9eccaace2d52164e648c2f81fb97f7bbe8"}
Oct 30 19:30:34.572252 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:34.571842    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-xqs7x"
Oct 30 19:30:34.591328 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:34.591299    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-xqs7x"
Oct 30 19:30:34.592754 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:34.592705    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-xqs7x" podStartSLOduration=1.934894814 podStartE2EDuration="18.592690719s" totalImagesPullingTime="16.657795905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:34.59168762 +0000 UTC m=+34.484301945" watchObservedRunningTime="2026-10-30 19:30:34.592690719 +0000 UTC m=+34.485305037"
Oct 30 19:30:38.486986 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:38.486957    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cf748dbd6-ljbz6"
Oct 30 19:30:38.544667 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:38.544630    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5fdc8795f9-cfh7p"]
Oct 30 19:30:41.377023 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.376844    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9" podUID="9e0ec109-980a-4203-b45b-25ebd01ae8ed" containerName="registry" containerID="cri-o://64496385aad5574c7eeede13b9922e403af1dd92a447852b95956ae876c8031e" gracePeriod=30
Oct 30 19:30:41.608441 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.608401    4823 generic.go:356] "Generic (PLEG): container finished" podID="9e0ec109-980a-4203-b45b-25ebd01ae8ed" containerID="64496385aad5574c7eeede13b9922e403af1dd92a447852b95956ae876c8031e" exitCode=0
Oct 30 19:30:41.608653 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.608527    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9" event={"ID":"9e0ec109-980a-4203-b45b-25ebd01ae8ed","Type":"ContainerDied","Data":"64496385aad5574c7eeede13b9922e403af1dd92a447852b95956ae876c8031e"}
Oct 30 19:30:41.681174 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.681139    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:41.843782 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.843739    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0ec109-980a-4203-b45b-25ebd01ae8ed-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e0ec109-980a-4203-b45b-25ebd01ae8ed-ca-trust-extracted\") pod \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") "
Oct 30 19:30:41.843973 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.843840    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-bound-sa-token\") pod \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") "
Oct 30 19:30:41.843973 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.843904    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-kube-api-access-6b8qf\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-kube-api-access-6b8qf\") pod \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") "
Oct 30 19:30:41.843973 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.843930    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-trusted-ca\") pod \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") "
Oct 30 19:30:41.843973 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.843960    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-certificates\") pod \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") "
Oct 30 19:30:41.844215 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.843994    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls\") pod \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") "
Oct 30 19:30:41.844215 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.844022    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-installation-pull-secrets\") pod \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") "
Oct 30 19:30:41.844215 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.844076    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-image-registry-private-configuration\") pod \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\" (UID: \"9e0ec109-980a-4203-b45b-25ebd01ae8ed\") "
Oct 30 19:30:41.844598 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.844563    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-certificates" pod "9e0ec109-980a-4203-b45b-25ebd01ae8ed" (UID: "9e0ec109-980a-4203-b45b-25ebd01ae8ed"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:41.844962 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.844938    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-trusted-ca" pod "9e0ec109-980a-4203-b45b-25ebd01ae8ed" (UID: "9e0ec109-980a-4203-b45b-25ebd01ae8ed"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:41.850461 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.850355    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-image-registry-private-configuration" pod "9e0ec109-980a-4203-b45b-25ebd01ae8ed" (UID: "9e0ec109-980a-4203-b45b-25ebd01ae8ed"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:41.850576 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.850535    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-installation-pull-secrets" pod "9e0ec109-980a-4203-b45b-25ebd01ae8ed" (UID: "9e0ec109-980a-4203-b45b-25ebd01ae8ed"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:41.852560 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.852535    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0ec109-980a-4203-b45b-25ebd01ae8ed-ca-trust-extracted" pod "9e0ec109-980a-4203-b45b-25ebd01ae8ed" (UID: "9e0ec109-980a-4203-b45b-25ebd01ae8ed"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:41.853581 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.853443    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-kube-api-access-6b8qf" pod "9e0ec109-980a-4203-b45b-25ebd01ae8ed" (UID: "9e0ec109-980a-4203-b45b-25ebd01ae8ed"). InnerVolumeSpecName "kube-api-access-6b8qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:41.853581 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.853501    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-bound-sa-token" pod "9e0ec109-980a-4203-b45b-25ebd01ae8ed" (UID: "9e0ec109-980a-4203-b45b-25ebd01ae8ed"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:41.853581 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.853512    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls" pod "9e0ec109-980a-4203-b45b-25ebd01ae8ed" (UID: "9e0ec109-980a-4203-b45b-25ebd01ae8ed"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:41.945659 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.945580    4823 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e0ec109-980a-4203-b45b-25ebd01ae8ed-ca-trust-extracted\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:41.945659 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.945619    4823 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-bound-sa-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:41.945659 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.945640    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b8qf\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-kube-api-access-6b8qf\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:41.945659 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.945654    4823 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-trusted-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:41.945659 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.945664    4823 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-certificates\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:41.946030 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.945673    4823 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e0ec109-980a-4203-b45b-25ebd01ae8ed-registry-tls\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:41.946030 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.945682    4823 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-installation-pull-secrets\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:41.946030 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:41.945693    4823 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e0ec109-980a-4203-b45b-25ebd01ae8ed-image-registry-private-configuration\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.620856 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:42.620823    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9"
Oct 30 19:30:42.621392 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:42.620819    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7ff684fc5d-nksx9" event={"ID":"9e0ec109-980a-4203-b45b-25ebd01ae8ed","Type":"ContainerDied","Data":"795d88aef4e175b414633a2097adc65c857beb6defcf3f81e840dd0715b93cfa"}
Oct 30 19:30:42.621392 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:42.620973    4823 scope.go:122] "RemoveContainer" containerID="64496385aad5574c7eeede13b9922e403af1dd92a447852b95956ae876c8031e"
Oct 30 19:30:42.638975 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:42.638950    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7ff684fc5d-nksx9"]
Oct 30 19:30:42.643069 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:42.643044    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7ff684fc5d-nksx9"]
Oct 30 19:30:44.276529 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:30:44.276496    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e0ec109-980a-4203-b45b-25ebd01ae8ed" path="/var/lib/kubelet/pods/9e0ec109-980a-4203-b45b-25ebd01ae8ed/volumes"
Oct 30 19:31:02.012583 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:31:02.012541    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:31:03.568874 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.568827    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p" podUID="153c1fd1-90a2-4063-b066-6503d796d6d8" containerName="registry" containerID="cri-o://af0167644ab6253695161cd09fbe1c2d98f0a9b26ee8b2d761a568e5e97478ab" gracePeriod=30
Oct 30 19:31:03.709894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.709852    4823 generic.go:356] "Generic (PLEG): container finished" podID="153c1fd1-90a2-4063-b066-6503d796d6d8" containerID="af0167644ab6253695161cd09fbe1c2d98f0a9b26ee8b2d761a568e5e97478ab" exitCode=0
Oct 30 19:31:03.710017 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.709908    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p" event={"ID":"153c1fd1-90a2-4063-b066-6503d796d6d8","Type":"ContainerDied","Data":"af0167644ab6253695161cd09fbe1c2d98f0a9b26ee8b2d761a568e5e97478ab"}
Oct 30 19:31:03.847590 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.847562    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:31:03.920830 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.920799    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-kube-api-access-zfdmm\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-kube-api-access-zfdmm\") pod \"153c1fd1-90a2-4063-b066-6503d796d6d8\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") "
Oct 30 19:31:03.921019 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.920848    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-trusted-ca\") pod \"153c1fd1-90a2-4063-b066-6503d796d6d8\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") "
Oct 30 19:31:03.921019 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.920908    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls\") pod \"153c1fd1-90a2-4063-b066-6503d796d6d8\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") "
Oct 30 19:31:03.921019 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.920940    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-image-registry-private-configuration\") pod \"153c1fd1-90a2-4063-b066-6503d796d6d8\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") "
Oct 30 19:31:03.921019 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.920972    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-installation-pull-secrets\") pod \"153c1fd1-90a2-4063-b066-6503d796d6d8\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") "
Oct 30 19:31:03.921019 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.920999    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-bound-sa-token\") pod \"153c1fd1-90a2-4063-b066-6503d796d6d8\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") "
Oct 30 19:31:03.921282 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.921057    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-certificates\") pod \"153c1fd1-90a2-4063-b066-6503d796d6d8\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") "
Oct 30 19:31:03.921282 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.921085    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153c1fd1-90a2-4063-b066-6503d796d6d8-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/153c1fd1-90a2-4063-b066-6503d796d6d8-ca-trust-extracted\") pod \"153c1fd1-90a2-4063-b066-6503d796d6d8\" (UID: \"153c1fd1-90a2-4063-b066-6503d796d6d8\") "
Oct 30 19:31:03.921550 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.921502    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-trusted-ca" pod "153c1fd1-90a2-4063-b066-6503d796d6d8" (UID: "153c1fd1-90a2-4063-b066-6503d796d6d8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:03.921907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.921880    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-certificates" pod "153c1fd1-90a2-4063-b066-6503d796d6d8" (UID: "153c1fd1-90a2-4063-b066-6503d796d6d8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:03.924057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.924008    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-installation-pull-secrets" pod "153c1fd1-90a2-4063-b066-6503d796d6d8" (UID: "153c1fd1-90a2-4063-b066-6503d796d6d8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:03.924057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.924037    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-kube-api-access-zfdmm" pod "153c1fd1-90a2-4063-b066-6503d796d6d8" (UID: "153c1fd1-90a2-4063-b066-6503d796d6d8"). InnerVolumeSpecName "kube-api-access-zfdmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:03.924621 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.924083    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-bound-sa-token" pod "153c1fd1-90a2-4063-b066-6503d796d6d8" (UID: "153c1fd1-90a2-4063-b066-6503d796d6d8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:03.924621 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.924303    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls" pod "153c1fd1-90a2-4063-b066-6503d796d6d8" (UID: "153c1fd1-90a2-4063-b066-6503d796d6d8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:03.924621 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.924357    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-image-registry-private-configuration" pod "153c1fd1-90a2-4063-b066-6503d796d6d8" (UID: "153c1fd1-90a2-4063-b066-6503d796d6d8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:03.928394 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:03.928374    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153c1fd1-90a2-4063-b066-6503d796d6d8-ca-trust-extracted" pod "153c1fd1-90a2-4063-b066-6503d796d6d8" (UID: "153c1fd1-90a2-4063-b066-6503d796d6d8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:04.021834 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:04.021803    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfdmm\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-kube-api-access-zfdmm\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:04.021834 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:04.021831    4823 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-trusted-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:04.021834 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:04.021841    4823 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-tls\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:04.022051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:04.021851    4823 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-image-registry-private-configuration\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:04.022051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:04.021878    4823 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/153c1fd1-90a2-4063-b066-6503d796d6d8-installation-pull-secrets\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:04.022051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:04.021887    4823 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/153c1fd1-90a2-4063-b066-6503d796d6d8-bound-sa-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:04.022051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:04.021896    4823 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/153c1fd1-90a2-4063-b066-6503d796d6d8-registry-certificates\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:04.022051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:04.021907    4823 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/153c1fd1-90a2-4063-b066-6503d796d6d8-ca-trust-extracted\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:04.714651 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:04.714623    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p"
Oct 30 19:31:04.714651 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:04.714639    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fdc8795f9-cfh7p" event={"ID":"153c1fd1-90a2-4063-b066-6503d796d6d8","Type":"ContainerDied","Data":"e51682e4319eef873d84ebeb6b159a14ea2a3400f45a680a73a7e1e11eb00605"}
Oct 30 19:31:04.715169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:04.714699    4823 scope.go:122] "RemoveContainer" containerID="af0167644ab6253695161cd09fbe1c2d98f0a9b26ee8b2d761a568e5e97478ab"
Oct 30 19:31:04.733794 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:04.733765    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5fdc8795f9-cfh7p"]
Oct 30 19:31:04.735750 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:04.735726    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5fdc8795f9-cfh7p"]
Oct 30 19:31:06.276657 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:06.276620    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153c1fd1-90a2-4063-b066-6503d796d6d8" path="/var/lib/kubelet/pods/153c1fd1-90a2-4063-b066-6503d796d6d8/volumes"
Oct 30 19:31:14.760673 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:14.760636    4823 generic.go:356] "Generic (PLEG): container finished" podID="6bc162c8-806a-4d1f-a046-3ade90069067" containerID="5769b290662579a73335b2e32a3793bd78ec56674eab6fa8df3897e2724ab13c" exitCode=0
Oct 30 19:31:14.761250 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:14.760714    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4vrds" event={"ID":"6bc162c8-806a-4d1f-a046-3ade90069067","Type":"ContainerDied","Data":"5769b290662579a73335b2e32a3793bd78ec56674eab6fa8df3897e2724ab13c"}
Oct 30 19:31:14.761250 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:14.761146    4823 scope.go:122] "RemoveContainer" containerID="5769b290662579a73335b2e32a3793bd78ec56674eab6fa8df3897e2724ab13c"
Oct 30 19:31:15.165259 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.165223    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-pfjv8-8n5cz"]
Oct 30 19:31:15.165475 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.165447    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-pfjv8-8n5cz" podUID="22a7299c-d539-47fa-87e2-626c5ca8bbbf" containerName="insights-gathering" containerID="cri-o://037c9e18e762d4c8bfb4e78b80594b52b506a9b16e449a1dfeffa95a699313de" gracePeriod=30
Oct 30 19:31:15.268101 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.268067    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-pbql9-qfd8c"]
Oct 30 19:31:15.268456 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.268394    4823 cpu_manager.go:414] "containerMap: removing container" podUID="153c1fd1-90a2-4063-b066-6503d796d6d8" containerName="registry"
Oct 30 19:31:15.268456 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.268405    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153c1fd1-90a2-4063-b066-6503d796d6d8" containerName="registry"
Oct 30 19:31:15.268456 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.268434    4823 cpu_manager.go:414] "containerMap: removing container" podUID="9e0ec109-980a-4203-b45b-25ebd01ae8ed" containerName="registry"
Oct 30 19:31:15.268456 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.268439    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0ec109-980a-4203-b45b-25ebd01ae8ed" containerName="registry"
Oct 30 19:31:15.268600 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.268488    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0ec109-980a-4203-b45b-25ebd01ae8ed" containerName="registry"
Oct 30 19:31:15.268600 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.268498    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="153c1fd1-90a2-4063-b066-6503d796d6d8" containerName="registry"
Oct 30 19:31:15.286883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.286839    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-pbql9-qfd8c"]
Oct 30 19:31:15.287008 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.286978    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pbql9-qfd8c"
Oct 30 19:31:15.417303 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.417275    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/136b7cd5-518d-45d6-9458-ec837933526e-service-ca-bundle\") pod \"periodic-gathering-pbql9-qfd8c\" (UID: \"136b7cd5-518d-45d6-9458-ec837933526e\") " pod="openshift-insights/periodic-gathering-pbql9-qfd8c"
Oct 30 19:31:15.417382 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.417335    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jx59\" (UniqueName: \"kubernetes.io/projected/136b7cd5-518d-45d6-9458-ec837933526e-kube-api-access-2jx59\") pod \"periodic-gathering-pbql9-qfd8c\" (UID: \"136b7cd5-518d-45d6-9458-ec837933526e\") " pod="openshift-insights/periodic-gathering-pbql9-qfd8c"
Oct 30 19:31:15.417458 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.417444    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/136b7cd5-518d-45d6-9458-ec837933526e-archives-path\") pod \"periodic-gathering-pbql9-qfd8c\" (UID: \"136b7cd5-518d-45d6-9458-ec837933526e\") " pod="openshift-insights/periodic-gathering-pbql9-qfd8c"
Oct 30 19:31:15.417495 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.417477    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/136b7cd5-518d-45d6-9458-ec837933526e-trusted-ca-bundle\") pod \"periodic-gathering-pbql9-qfd8c\" (UID: \"136b7cd5-518d-45d6-9458-ec837933526e\") " pod="openshift-insights/periodic-gathering-pbql9-qfd8c"
Oct 30 19:31:15.450153 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.450131    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:31:15.518340 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.518314    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-trusted-ca-bundle\") pod \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") "
Oct 30 19:31:15.518460 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.518349    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-service-ca-bundle\") pod \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") "
Oct 30 19:31:15.518460 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.518386    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a7299c-d539-47fa-87e2-626c5ca8bbbf-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/22a7299c-d539-47fa-87e2-626c5ca8bbbf-archives-path\") pod \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") "
Oct 30 19:31:15.518545 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.518513    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a7299c-d539-47fa-87e2-626c5ca8bbbf-kube-api-access-g2svq\" (UniqueName: \"kubernetes.io/projected/22a7299c-d539-47fa-87e2-626c5ca8bbbf-kube-api-access-g2svq\") pod \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\" (UID: \"22a7299c-d539-47fa-87e2-626c5ca8bbbf\") "
Oct 30 19:31:15.518638 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.518618    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/136b7cd5-518d-45d6-9458-ec837933526e-service-ca-bundle\") pod \"periodic-gathering-pbql9-qfd8c\" (UID: \"136b7cd5-518d-45d6-9458-ec837933526e\") " pod="openshift-insights/periodic-gathering-pbql9-qfd8c"
Oct 30 19:31:15.518736 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.518697    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a7299c-d539-47fa-87e2-626c5ca8bbbf-archives-path" pod "22a7299c-d539-47fa-87e2-626c5ca8bbbf" (UID: "22a7299c-d539-47fa-87e2-626c5ca8bbbf"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:15.518736 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.518715    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jx59\" (UniqueName: \"kubernetes.io/projected/136b7cd5-518d-45d6-9458-ec837933526e-kube-api-access-2jx59\") pod \"periodic-gathering-pbql9-qfd8c\" (UID: \"136b7cd5-518d-45d6-9458-ec837933526e\") " pod="openshift-insights/periodic-gathering-pbql9-qfd8c"
Oct 30 19:31:15.518878 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.518703    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-service-ca-bundle" pod "22a7299c-d539-47fa-87e2-626c5ca8bbbf" (UID: "22a7299c-d539-47fa-87e2-626c5ca8bbbf"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:15.518878 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.518806    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/136b7cd5-518d-45d6-9458-ec837933526e-archives-path\") pod \"periodic-gathering-pbql9-qfd8c\" (UID: \"136b7cd5-518d-45d6-9458-ec837933526e\") " pod="openshift-insights/periodic-gathering-pbql9-qfd8c"
Oct 30 19:31:15.518878 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.518723    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-trusted-ca-bundle" pod "22a7299c-d539-47fa-87e2-626c5ca8bbbf" (UID: "22a7299c-d539-47fa-87e2-626c5ca8bbbf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:15.518997 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.518882    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/136b7cd5-518d-45d6-9458-ec837933526e-trusted-ca-bundle\") pod \"periodic-gathering-pbql9-qfd8c\" (UID: \"136b7cd5-518d-45d6-9458-ec837933526e\") " pod="openshift-insights/periodic-gathering-pbql9-qfd8c"
Oct 30 19:31:15.518997 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.518929    4823 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-trusted-ca-bundle\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:15.518997 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.518944    4823 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22a7299c-d539-47fa-87e2-626c5ca8bbbf-service-ca-bundle\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:15.518997 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.518958    4823 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/22a7299c-d539-47fa-87e2-626c5ca8bbbf-archives-path\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:15.519118 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.519095    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/136b7cd5-518d-45d6-9458-ec837933526e-archives-path\") pod \"periodic-gathering-pbql9-qfd8c\" (UID: \"136b7cd5-518d-45d6-9458-ec837933526e\") " pod="openshift-insights/periodic-gathering-pbql9-qfd8c"
Oct 30 19:31:15.519317 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.519301    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/136b7cd5-518d-45d6-9458-ec837933526e-service-ca-bundle\") pod \"periodic-gathering-pbql9-qfd8c\" (UID: \"136b7cd5-518d-45d6-9458-ec837933526e\") " pod="openshift-insights/periodic-gathering-pbql9-qfd8c"
Oct 30 19:31:15.519693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.519668    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/136b7cd5-518d-45d6-9458-ec837933526e-trusted-ca-bundle\") pod \"periodic-gathering-pbql9-qfd8c\" (UID: \"136b7cd5-518d-45d6-9458-ec837933526e\") " pod="openshift-insights/periodic-gathering-pbql9-qfd8c"
Oct 30 19:31:15.520989 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.520967    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a7299c-d539-47fa-87e2-626c5ca8bbbf-kube-api-access-g2svq" pod "22a7299c-d539-47fa-87e2-626c5ca8bbbf" (UID: "22a7299c-d539-47fa-87e2-626c5ca8bbbf"). InnerVolumeSpecName "kube-api-access-g2svq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:15.530831 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.530812    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jx59\" (UniqueName: \"kubernetes.io/projected/136b7cd5-518d-45d6-9458-ec837933526e-kube-api-access-2jx59\") pod \"periodic-gathering-pbql9-qfd8c\" (UID: \"136b7cd5-518d-45d6-9458-ec837933526e\") " pod="openshift-insights/periodic-gathering-pbql9-qfd8c"
Oct 30 19:31:15.616370 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.616345    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pbql9-qfd8c"
Oct 30 19:31:15.619672 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.619651    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2svq\" (UniqueName: \"kubernetes.io/projected/22a7299c-d539-47fa-87e2-626c5ca8bbbf-kube-api-access-g2svq\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:15.766194 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.766158    4823 generic.go:356] "Generic (PLEG): container finished" podID="22a7299c-d539-47fa-87e2-626c5ca8bbbf" containerID="037c9e18e762d4c8bfb4e78b80594b52b506a9b16e449a1dfeffa95a699313de" exitCode=2
Oct 30 19:31:15.766698 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.766218    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pfjv8-8n5cz"
Oct 30 19:31:15.766698 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.766248    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pfjv8-8n5cz" event={"ID":"22a7299c-d539-47fa-87e2-626c5ca8bbbf","Type":"ContainerDied","Data":"037c9e18e762d4c8bfb4e78b80594b52b506a9b16e449a1dfeffa95a699313de"}
Oct 30 19:31:15.766698 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.766296    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pfjv8-8n5cz" event={"ID":"22a7299c-d539-47fa-87e2-626c5ca8bbbf","Type":"ContainerDied","Data":"33c7fd5ad315ac4b5b970c81af2a9c37e116079a5627799aa4c7c52b794ed9df"}
Oct 30 19:31:15.766698 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.766317    4823 scope.go:122] "RemoveContainer" containerID="037c9e18e762d4c8bfb4e78b80594b52b506a9b16e449a1dfeffa95a699313de"
Oct 30 19:31:15.768320 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.768297    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4vrds" event={"ID":"6bc162c8-806a-4d1f-a046-3ade90069067","Type":"ContainerStarted","Data":"2001ea102651840fbebd44028b038d29fdbc8403408b245dddd5d3cc9de87d8f"}
Oct 30 19:31:15.779977 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.779956    4823 scope.go:122] "RemoveContainer" containerID="037c9e18e762d4c8bfb4e78b80594b52b506a9b16e449a1dfeffa95a699313de"
Oct 30 19:31:15.780263 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:31:15.780234    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037c9e18e762d4c8bfb4e78b80594b52b506a9b16e449a1dfeffa95a699313de\": container with ID starting with 037c9e18e762d4c8bfb4e78b80594b52b506a9b16e449a1dfeffa95a699313de not found: ID does not exist" containerID="037c9e18e762d4c8bfb4e78b80594b52b506a9b16e449a1dfeffa95a699313de"
Oct 30 19:31:15.780327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.780272    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037c9e18e762d4c8bfb4e78b80594b52b506a9b16e449a1dfeffa95a699313de"} err="failed to get container status \"037c9e18e762d4c8bfb4e78b80594b52b506a9b16e449a1dfeffa95a699313de\": rpc error: code = NotFound desc = could not find container \"037c9e18e762d4c8bfb4e78b80594b52b506a9b16e449a1dfeffa95a699313de\": container with ID starting with 037c9e18e762d4c8bfb4e78b80594b52b506a9b16e449a1dfeffa95a699313de not found: ID does not exist"
Oct 30 19:31:15.806078 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.806052    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-pfjv8-8n5cz"]
Oct 30 19:31:15.809657 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.809635    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-pfjv8-8n5cz"]
Oct 30 19:31:15.918225 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:15.918189    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-pbql9-qfd8c"]
Oct 30 19:31:15.921390 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:31:15.921354    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136b7cd5_518d_45d6_9458_ec837933526e.slice/crio-94c5c329ae5594564dca4761b659fea6af7fd0b1ee163f18d3eb5e59c7a05571 WatchSource:0}: Error finding container 94c5c329ae5594564dca4761b659fea6af7fd0b1ee163f18d3eb5e59c7a05571: Status 404 returned error can't find the container with id 94c5c329ae5594564dca4761b659fea6af7fd0b1ee163f18d3eb5e59c7a05571
Oct 30 19:31:16.276316 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:16.276245    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a7299c-d539-47fa-87e2-626c5ca8bbbf" path="/var/lib/kubelet/pods/22a7299c-d539-47fa-87e2-626c5ca8bbbf/volumes"
Oct 30 19:31:16.311089 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:16.311059    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cb4fdcd5-zsxtw_df3b6a17-4bbf-41dc-83e5-df5172b87b7f/router/0.log"
Oct 30 19:31:16.368247 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:16.368212    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vzpfb_3615ac3b-837c-46ea-b575-4cdc1edc78a0/serve-healthcheck-canary/0.log"
Oct 30 19:31:16.773760 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:16.773724    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pbql9-qfd8c" event={"ID":"136b7cd5-518d-45d6-9458-ec837933526e","Type":"ContainerStarted","Data":"d4af8ee906159cd84e46bcb36623a027468e9182b30351b0eea4d63343e5af5a"}
Oct 30 19:31:16.774280 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:16.773782    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pbql9-qfd8c" event={"ID":"136b7cd5-518d-45d6-9458-ec837933526e","Type":"ContainerStarted","Data":"94c5c329ae5594564dca4761b659fea6af7fd0b1ee163f18d3eb5e59c7a05571"}
Oct 30 19:31:16.792152 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:16.792108    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-pbql9-qfd8c" podStartSLOduration=1.792095333 podStartE2EDuration="1.792095333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:16.791667877 +0000 UTC m=+76.684282201" watchObservedRunningTime="2026-10-30 19:31:16.792095333 +0000 UTC m=+76.684709656"
Oct 30 19:31:29.828154 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:29.828114    4823 generic.go:356] "Generic (PLEG): container finished" podID="468ec5d6-34c5-4577-84a5-ec661a3f2736" containerID="dc7567f9a92d899a1b5c7dda09cd1f370124a0b59768c649797c40c8e8d8bb38" exitCode=0
Oct 30 19:31:29.828565 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:29.828185    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv" event={"ID":"468ec5d6-34c5-4577-84a5-ec661a3f2736","Type":"ContainerDied","Data":"dc7567f9a92d899a1b5c7dda09cd1f370124a0b59768c649797c40c8e8d8bb38"}
Oct 30 19:31:29.828565 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:29.828515    4823 scope.go:122] "RemoveContainer" containerID="dc7567f9a92d899a1b5c7dda09cd1f370124a0b59768c649797c40c8e8d8bb38"
Oct 30 19:31:30.838815 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:31:30.838777    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d2pkv" event={"ID":"468ec5d6-34c5-4577-84a5-ec661a3f2736","Type":"ContainerStarted","Data":"ae67637668b91cacdcfd6b57d2b80a1fbf9e48dd955d4fe76dcfa1b88043a1c3"}
Oct 30 19:31:32.013315 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:31:32.013275    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:32:02.012229 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:32:02.012183    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:32:32.012380 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:32:32.012300    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:32:41.133944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:41.133908    4823 generic.go:356] "Generic (PLEG): container finished" podID="136b7cd5-518d-45d6-9458-ec837933526e" containerID="d4af8ee906159cd84e46bcb36623a027468e9182b30351b0eea4d63343e5af5a" exitCode=0
Oct 30 19:32:41.134324 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:41.133977    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pbql9-qfd8c" event={"ID":"136b7cd5-518d-45d6-9458-ec837933526e","Type":"ContainerDied","Data":"d4af8ee906159cd84e46bcb36623a027468e9182b30351b0eea4d63343e5af5a"}
Oct 30 19:32:42.265587 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:42.265565    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pbql9-qfd8c"
Oct 30 19:32:42.434457 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:42.434382    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136b7cd5-518d-45d6-9458-ec837933526e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/136b7cd5-518d-45d6-9458-ec837933526e-trusted-ca-bundle\") pod \"136b7cd5-518d-45d6-9458-ec837933526e\" (UID: \"136b7cd5-518d-45d6-9458-ec837933526e\") "
Oct 30 19:32:42.434457 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:42.434422    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136b7cd5-518d-45d6-9458-ec837933526e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/136b7cd5-518d-45d6-9458-ec837933526e-archives-path\") pod \"136b7cd5-518d-45d6-9458-ec837933526e\" (UID: \"136b7cd5-518d-45d6-9458-ec837933526e\") "
Oct 30 19:32:42.434640 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:42.434494    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136b7cd5-518d-45d6-9458-ec837933526e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/136b7cd5-518d-45d6-9458-ec837933526e-service-ca-bundle\") pod \"136b7cd5-518d-45d6-9458-ec837933526e\" (UID: \"136b7cd5-518d-45d6-9458-ec837933526e\") "
Oct 30 19:32:42.434640 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:42.434518    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136b7cd5-518d-45d6-9458-ec837933526e-kube-api-access-2jx59\" (UniqueName: \"kubernetes.io/projected/136b7cd5-518d-45d6-9458-ec837933526e-kube-api-access-2jx59\") pod \"136b7cd5-518d-45d6-9458-ec837933526e\" (UID: \"136b7cd5-518d-45d6-9458-ec837933526e\") "
Oct 30 19:32:42.435073 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:42.434786    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136b7cd5-518d-45d6-9458-ec837933526e-trusted-ca-bundle" pod "136b7cd5-518d-45d6-9458-ec837933526e" (UID: "136b7cd5-518d-45d6-9458-ec837933526e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:42.435073 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:42.434811    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136b7cd5-518d-45d6-9458-ec837933526e-archives-path" pod "136b7cd5-518d-45d6-9458-ec837933526e" (UID: "136b7cd5-518d-45d6-9458-ec837933526e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:42.435073 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:42.434827    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136b7cd5-518d-45d6-9458-ec837933526e-service-ca-bundle" pod "136b7cd5-518d-45d6-9458-ec837933526e" (UID: "136b7cd5-518d-45d6-9458-ec837933526e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:42.436623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:42.436599    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136b7cd5-518d-45d6-9458-ec837933526e-kube-api-access-2jx59" pod "136b7cd5-518d-45d6-9458-ec837933526e" (UID: "136b7cd5-518d-45d6-9458-ec837933526e"). InnerVolumeSpecName "kube-api-access-2jx59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:42.535373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:42.535334    4823 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/136b7cd5-518d-45d6-9458-ec837933526e-trusted-ca-bundle\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:42.535373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:42.535366    4823 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/136b7cd5-518d-45d6-9458-ec837933526e-archives-path\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:42.535373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:42.535375    4823 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/136b7cd5-518d-45d6-9458-ec837933526e-service-ca-bundle\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:42.535601 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:42.535386    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jx59\" (UniqueName: \"kubernetes.io/projected/136b7cd5-518d-45d6-9458-ec837933526e-kube-api-access-2jx59\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:43.143287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:43.143257    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pbql9-qfd8c"
Oct 30 19:32:43.143287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:43.143264    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pbql9-qfd8c" event={"ID":"136b7cd5-518d-45d6-9458-ec837933526e","Type":"ContainerDied","Data":"94c5c329ae5594564dca4761b659fea6af7fd0b1ee163f18d3eb5e59c7a05571"}
Oct 30 19:32:43.143545 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:32:43.143303    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94c5c329ae5594564dca4761b659fea6af7fd0b1ee163f18d3eb5e59c7a05571"
Oct 30 19:33:02.012358 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:33:02.012317    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:33:32.013272 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:33:32.013234    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:33:54.984266 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:54.984231    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"]
Oct 30 19:33:54.984790 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:54.984700    4823 cpu_manager.go:414] "containerMap: removing container" podUID="22a7299c-d539-47fa-87e2-626c5ca8bbbf" containerName="insights-gathering"
Oct 30 19:33:54.984790 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:54.984721    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a7299c-d539-47fa-87e2-626c5ca8bbbf" containerName="insights-gathering"
Oct 30 19:33:54.984790 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:54.984741    4823 cpu_manager.go:414] "containerMap: removing container" podUID="136b7cd5-518d-45d6-9458-ec837933526e" containerName="insights-gathering"
Oct 30 19:33:54.984790 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:54.984749    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136b7cd5-518d-45d6-9458-ec837933526e" containerName="insights-gathering"
Oct 30 19:33:54.985018 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:54.984843    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a7299c-d539-47fa-87e2-626c5ca8bbbf" containerName="insights-gathering"
Oct 30 19:33:54.985018 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:54.984884    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="136b7cd5-518d-45d6-9458-ec837933526e" containerName="insights-gathering"
Oct 30 19:33:54.986941 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:54.986920    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:54.993016 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:54.992999    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:33:54.993173 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:54.993010    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fbh7r\""
Oct 30 19:33:55.000411 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.000389    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"]
Oct 30 19:33:55.043117 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.043092    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892f7495-a9e2-489d-92c5-12f35e1e65f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.043251 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.043123    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892f7495-a9e2-489d-92c5-12f35e1e65f3-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.043251 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.043157    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc99k\" (UniqueName: \"kubernetes.io/projected/892f7495-a9e2-489d-92c5-12f35e1e65f3-kube-api-access-zc99k\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.043370 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.043293    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892f7495-a9e2-489d-92c5-12f35e1e65f3-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.043370 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.043360    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892f7495-a9e2-489d-92c5-12f35e1e65f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.043450 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.043429    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892f7495-a9e2-489d-92c5-12f35e1e65f3-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.043495 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.043450    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892f7495-a9e2-489d-92c5-12f35e1e65f3-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.043495 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.043467    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892f7495-a9e2-489d-92c5-12f35e1e65f3-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.043598 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.043582    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892f7495-a9e2-489d-92c5-12f35e1e65f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.090735 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.090705    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"]
Oct 30 19:33:55.093317 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.093295    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.100346 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.100316    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-l4zbk\""
Oct 30 19:33:55.107576 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.107553    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"]
Oct 30 19:33:55.144889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.144845    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892f7495-a9e2-489d-92c5-12f35e1e65f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.145029 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.144912    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892f7495-a9e2-489d-92c5-12f35e1e65f3-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.145029 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.144932    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892f7495-a9e2-489d-92c5-12f35e1e65f3-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.145029 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.144948    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892f7495-a9e2-489d-92c5-12f35e1e65f3-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.145029 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.144977    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.145232 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145109    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.145232 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145209    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rcsm\" (UniqueName: \"kubernetes.io/projected/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-kube-api-access-2rcsm\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.145334 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145300    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.145395 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145339    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892f7495-a9e2-489d-92c5-12f35e1e65f3-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.145395 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145341    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892f7495-a9e2-489d-92c5-12f35e1e65f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.145395 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145387    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.145527 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145402    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892f7495-a9e2-489d-92c5-12f35e1e65f3-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.145527 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145424    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892f7495-a9e2-489d-92c5-12f35e1e65f3-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.145527 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145485    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.145636 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145538    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892f7495-a9e2-489d-92c5-12f35e1e65f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.145636 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145619    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892f7495-a9e2-489d-92c5-12f35e1e65f3-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.145718 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145657    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.145718 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145695    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.145841 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145819    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc99k\" (UniqueName: \"kubernetes.io/projected/892f7495-a9e2-489d-92c5-12f35e1e65f3-kube-api-access-zc99k\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.145930 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145884    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892f7495-a9e2-489d-92c5-12f35e1e65f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.145930 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145897    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.146030 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.145934    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892f7495-a9e2-489d-92c5-12f35e1e65f3-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.146232 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.146213    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892f7495-a9e2-489d-92c5-12f35e1e65f3-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.147694 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.147673    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892f7495-a9e2-489d-92c5-12f35e1e65f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.147770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.147682    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892f7495-a9e2-489d-92c5-12f35e1e65f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.157420 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.157389    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892f7495-a9e2-489d-92c5-12f35e1e65f3-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.157501 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.157442    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc99k\" (UniqueName: \"kubernetes.io/projected/892f7495-a9e2-489d-92c5-12f35e1e65f3-kube-api-access-zc99k\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6p5n6\" (UID: \"892f7495-a9e2-489d-92c5-12f35e1e65f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.246706 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.246628    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.246706 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.246665    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rcsm\" (UniqueName: \"kubernetes.io/projected/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-kube-api-access-2rcsm\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.246924 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.246792    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.246924 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.246831    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.246924 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.246883    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.246924 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.246922    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.247157 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.247005    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.247157 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.247016    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.247157 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.247073    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.247157 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.247143    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.247362 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.247191    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.247362 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.247224    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.247475 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.247445    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.247518 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.247499    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.249028 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.249007    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.249356 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.249336    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.258939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.258919    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.259266 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.259246    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rcsm\" (UniqueName: \"kubernetes.io/projected/f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac-kube-api-access-2rcsm\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k\" (UID: \"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.300253 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.300227    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:33:55.407139 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.407109    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:33:55.578340 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.578271    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"]
Oct 30 19:33:55.581601 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:33:55.581569    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892f7495_a9e2_489d_92c5_12f35e1e65f3.slice/crio-9b0992cb9eb81864f168733cc2c15063083e1cffc190b7367020b7daddb92cd8 WatchSource:0}: Error finding container 9b0992cb9eb81864f168733cc2c15063083e1cffc190b7367020b7daddb92cd8: Status 404 returned error can't find the container with id 9b0992cb9eb81864f168733cc2c15063083e1cffc190b7367020b7daddb92cd8
Oct 30 19:33:55.688211 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:55.688164    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"]
Oct 30 19:33:55.690961 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:33:55.690934    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c3a3c0_a2ba_43ad_a083_ed7f6b0ff6ac.slice/crio-2523add84d177c3b70dacec854eb1d68d3b460f58b6c0239514286e09ab76f81 WatchSource:0}: Error finding container 2523add84d177c3b70dacec854eb1d68d3b460f58b6c0239514286e09ab76f81: Status 404 returned error can't find the container with id 2523add84d177c3b70dacec854eb1d68d3b460f58b6c0239514286e09ab76f81
Oct 30 19:33:56.425127 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:56.425092    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k" event={"ID":"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac","Type":"ContainerStarted","Data":"2523add84d177c3b70dacec854eb1d68d3b460f58b6c0239514286e09ab76f81"}
Oct 30 19:33:56.426243 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:56.426220    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6" event={"ID":"892f7495-a9e2-489d-92c5-12f35e1e65f3","Type":"ContainerStarted","Data":"9b0992cb9eb81864f168733cc2c15063083e1cffc190b7367020b7daddb92cd8"}
Oct 30 19:33:59.056370 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.056336    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:33:59.056648 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.056420    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:33:59.056648 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.056452    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:33:59.091191 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.091169    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj"]
Oct 30 19:33:59.097555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.097536    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj"
Oct 30 19:33:59.104271 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.104247    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj"]
Oct 30 19:33:59.105153 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.105135    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 19:33:59.105261 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.105167    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 19:33:59.105261 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.105210    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-qvslp\""
Oct 30 19:33:59.141585 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.141543    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:33:59.141688 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.141636    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:33:59.141733 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.141696    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:33:59.181380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.181347    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1429b742-e007-48c8-9c63-d77f4d7c9ec4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj\" (UID: \"1429b742-e007-48c8-9c63-d77f4d7c9ec4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj"
Oct 30 19:33:59.181551 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.181388    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1429b742-e007-48c8-9c63-d77f4d7c9ec4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj\" (UID: \"1429b742-e007-48c8-9c63-d77f4d7c9ec4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj"
Oct 30 19:33:59.181551 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.181415    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt9t5\" (UniqueName: \"kubernetes.io/projected/1429b742-e007-48c8-9c63-d77f4d7c9ec4-kube-api-access-jt9t5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj\" (UID: \"1429b742-e007-48c8-9c63-d77f4d7c9ec4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj"
Oct 30 19:33:59.282665 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.282620    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1429b742-e007-48c8-9c63-d77f4d7c9ec4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj\" (UID: \"1429b742-e007-48c8-9c63-d77f4d7c9ec4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj"
Oct 30 19:33:59.282830 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.282671    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1429b742-e007-48c8-9c63-d77f4d7c9ec4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj\" (UID: \"1429b742-e007-48c8-9c63-d77f4d7c9ec4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj"
Oct 30 19:33:59.282830 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.282701    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt9t5\" (UniqueName: \"kubernetes.io/projected/1429b742-e007-48c8-9c63-d77f4d7c9ec4-kube-api-access-jt9t5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj\" (UID: \"1429b742-e007-48c8-9c63-d77f4d7c9ec4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj"
Oct 30 19:33:59.283093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.283068    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1429b742-e007-48c8-9c63-d77f4d7c9ec4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj\" (UID: \"1429b742-e007-48c8-9c63-d77f4d7c9ec4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj"
Oct 30 19:33:59.283093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.283085    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1429b742-e007-48c8-9c63-d77f4d7c9ec4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj\" (UID: \"1429b742-e007-48c8-9c63-d77f4d7c9ec4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj"
Oct 30 19:33:59.295151 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.295124    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt9t5\" (UniqueName: \"kubernetes.io/projected/1429b742-e007-48c8-9c63-d77f4d7c9ec4-kube-api-access-jt9t5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj\" (UID: \"1429b742-e007-48c8-9c63-d77f4d7c9ec4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj"
Oct 30 19:33:59.426318 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.426291    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj"
Oct 30 19:33:59.443920 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.443881    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k" event={"ID":"f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac","Type":"ContainerStarted","Data":"d41de5a12ebc67917bd75ff198bf41b4dc5eb1f47f29c16a4526c9ce7b2ea1ae"}
Oct 30 19:33:59.445623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.445591    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6" event={"ID":"892f7495-a9e2-489d-92c5-12f35e1e65f3","Type":"ContainerStarted","Data":"0945026cbe09b6d844248ef4720c838b050a20ad9e4917fb079c1d1c5b6f12e8"}
Oct 30 19:33:59.468996 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.468690    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k" podStartSLOduration=1.02068112 podStartE2EDuration="4.468671291s" totalImagesPullingTime="3.447990171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:59.466665628 +0000 UTC m=+239.359279952" watchObservedRunningTime="2026-10-30 19:33:59.468671291 +0000 UTC m=+239.361285616"
Oct 30 19:33:59.491509 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.490950    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6" podStartSLOduration=2.018959616 podStartE2EDuration="5.490929095s" totalImagesPullingTime="3.471969479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:59.489296913 +0000 UTC m=+239.381911233" watchObservedRunningTime="2026-10-30 19:33:59.490929095 +0000 UTC m=+239.383543420"
Oct 30 19:33:59.707946 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:33:59.707852    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj"]
Oct 30 19:33:59.712938 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:33:59.712901    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1429b742_e007_48c8_9c63_d77f4d7c9ec4.slice/crio-8bf543ed5065c727d90df2d8ef5c35e7681a5dd21b851f82797646f761a093d4 WatchSource:0}: Error finding container 8bf543ed5065c727d90df2d8ef5c35e7681a5dd21b851f82797646f761a093d4: Status 404 returned error can't find the container with id 8bf543ed5065c727d90df2d8ef5c35e7681a5dd21b851f82797646f761a093d4
Oct 30 19:34:00.300839 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:00.300809    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:34:00.306505 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:00.306479    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:34:00.407792 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:00.407756    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:34:00.413773 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:00.413749    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:34:00.451741 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:00.451702    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj" event={"ID":"1429b742-e007-48c8-9c63-d77f4d7c9ec4","Type":"ContainerStarted","Data":"8bf543ed5065c727d90df2d8ef5c35e7681a5dd21b851f82797646f761a093d4"}
Oct 30 19:34:00.452404 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:00.452378    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:34:00.452529 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:00.452426    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:34:00.453283 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:00.453203    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k"
Oct 30 19:34:00.453404 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:00.453382    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:34:02.017952 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:34:02.017908    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:34:05.476807 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:05.476769    4823 generic.go:356] "Generic (PLEG): container finished" podID="1429b742-e007-48c8-9c63-d77f4d7c9ec4" containerID="517b8e05785a0ed199a3051b21603ce9cc46513222f54fc018ef9848fa1e2308" exitCode=0
Oct 30 19:34:05.477231 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:05.476833    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj" event={"ID":"1429b742-e007-48c8-9c63-d77f4d7c9ec4","Type":"ContainerDied","Data":"517b8e05785a0ed199a3051b21603ce9cc46513222f54fc018ef9848fa1e2308"}
Oct 30 19:34:06.482945 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:06.482917    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj" event={"ID":"1429b742-e007-48c8-9c63-d77f4d7c9ec4","Type":"ContainerStarted","Data":"f82e5222686666826f7e8b6aa478c5bbb42f18384dfebab701e4682c9a2f7e00"}
Oct 30 19:34:07.489102 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:07.489066    4823 generic.go:356] "Generic (PLEG): container finished" podID="1429b742-e007-48c8-9c63-d77f4d7c9ec4" containerID="f82e5222686666826f7e8b6aa478c5bbb42f18384dfebab701e4682c9a2f7e00" exitCode=0
Oct 30 19:34:07.489555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:07.489150    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj" event={"ID":"1429b742-e007-48c8-9c63-d77f4d7c9ec4","Type":"ContainerDied","Data":"f82e5222686666826f7e8b6aa478c5bbb42f18384dfebab701e4682c9a2f7e00"}
Oct 30 19:34:13.519771 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:13.519680    4823 generic.go:356] "Generic (PLEG): container finished" podID="1429b742-e007-48c8-9c63-d77f4d7c9ec4" containerID="b29674cd52537cfb1bef7a72213bc0b35177684cb1083a395c40f0a047654e9a" exitCode=0
Oct 30 19:34:13.519771 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:13.519755    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj" event={"ID":"1429b742-e007-48c8-9c63-d77f4d7c9ec4","Type":"ContainerDied","Data":"b29674cd52537cfb1bef7a72213bc0b35177684cb1083a395c40f0a047654e9a"}
Oct 30 19:34:14.648069 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:14.648046    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj"
Oct 30 19:34:14.729727 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:14.729683    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1429b742-e007-48c8-9c63-d77f4d7c9ec4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1429b742-e007-48c8-9c63-d77f4d7c9ec4-bundle\") pod \"1429b742-e007-48c8-9c63-d77f4d7c9ec4\" (UID: \"1429b742-e007-48c8-9c63-d77f4d7c9ec4\") "
Oct 30 19:34:14.729727 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:14.729739    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1429b742-e007-48c8-9c63-d77f4d7c9ec4-kube-api-access-jt9t5\" (UniqueName: \"kubernetes.io/projected/1429b742-e007-48c8-9c63-d77f4d7c9ec4-kube-api-access-jt9t5\") pod \"1429b742-e007-48c8-9c63-d77f4d7c9ec4\" (UID: \"1429b742-e007-48c8-9c63-d77f4d7c9ec4\") "
Oct 30 19:34:14.729964 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:14.729805    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1429b742-e007-48c8-9c63-d77f4d7c9ec4-util\" (UniqueName: \"kubernetes.io/empty-dir/1429b742-e007-48c8-9c63-d77f4d7c9ec4-util\") pod \"1429b742-e007-48c8-9c63-d77f4d7c9ec4\" (UID: \"1429b742-e007-48c8-9c63-d77f4d7c9ec4\") "
Oct 30 19:34:14.730218 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:14.730193    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1429b742-e007-48c8-9c63-d77f4d7c9ec4-bundle" pod "1429b742-e007-48c8-9c63-d77f4d7c9ec4" (UID: "1429b742-e007-48c8-9c63-d77f4d7c9ec4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:14.732112 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:14.732086    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1429b742-e007-48c8-9c63-d77f4d7c9ec4-kube-api-access-jt9t5" pod "1429b742-e007-48c8-9c63-d77f4d7c9ec4" (UID: "1429b742-e007-48c8-9c63-d77f4d7c9ec4"). InnerVolumeSpecName "kube-api-access-jt9t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:14.734037 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:14.734009    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1429b742-e007-48c8-9c63-d77f4d7c9ec4-util" pod "1429b742-e007-48c8-9c63-d77f4d7c9ec4" (UID: "1429b742-e007-48c8-9c63-d77f4d7c9ec4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:14.830452 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:14.830381    4823 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1429b742-e007-48c8-9c63-d77f4d7c9ec4-bundle\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:14.830452 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:14.830406    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt9t5\" (UniqueName: \"kubernetes.io/projected/1429b742-e007-48c8-9c63-d77f4d7c9ec4-kube-api-access-jt9t5\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:14.830452 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:14.830416    4823 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1429b742-e007-48c8-9c63-d77f4d7c9ec4-util\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:15.529566 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:15.529540    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj"
Oct 30 19:34:15.529728 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:15.529535    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vg7kj" event={"ID":"1429b742-e007-48c8-9c63-d77f4d7c9ec4","Type":"ContainerDied","Data":"8bf543ed5065c727d90df2d8ef5c35e7681a5dd21b851f82797646f761a093d4"}
Oct 30 19:34:15.529728 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:15.529655    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bf543ed5065c727d90df2d8ef5c35e7681a5dd21b851f82797646f761a093d4"
Oct 30 19:34:32.012992 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:34:32.012951    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:34:34.861010 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.860975    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"]
Oct 30 19:34:34.861546 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.861325    4823 cpu_manager.go:414] "containerMap: removing container" podUID="1429b742-e007-48c8-9c63-d77f4d7c9ec4" containerName="util"
Oct 30 19:34:34.861546 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.861337    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1429b742-e007-48c8-9c63-d77f4d7c9ec4" containerName="util"
Oct 30 19:34:34.861546 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.861355    4823 cpu_manager.go:414] "containerMap: removing container" podUID="1429b742-e007-48c8-9c63-d77f4d7c9ec4" containerName="extract"
Oct 30 19:34:34.861546 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.861360    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1429b742-e007-48c8-9c63-d77f4d7c9ec4" containerName="extract"
Oct 30 19:34:34.861546 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.861367    4823 cpu_manager.go:414] "containerMap: removing container" podUID="1429b742-e007-48c8-9c63-d77f4d7c9ec4" containerName="pull"
Oct 30 19:34:34.861546 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.861372    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1429b742-e007-48c8-9c63-d77f4d7c9ec4" containerName="pull"
Oct 30 19:34:34.861546 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.861428    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="1429b742-e007-48c8-9c63-d77f4d7c9ec4" containerName="extract"
Oct 30 19:34:34.863497 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.863468    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.884599 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.884564    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2504d4c-a07f-4219-8c6b-948cc22da563-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.884599 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.884604    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2504d4c-a07f-4219-8c6b-948cc22da563-tmp\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.884808 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.884631    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2504d4c-a07f-4219-8c6b-948cc22da563-manifests\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.884900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.884842    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2504d4c-a07f-4219-8c6b-948cc22da563-webhook-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.885337 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.885320    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmjdw\" (UniqueName: \"kubernetes.io/projected/c2504d4c-a07f-4219-8c6b-948cc22da563-kube-api-access-hmjdw\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.885391 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.885354    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2504d4c-a07f-4219-8c6b-948cc22da563-metrics-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.914081 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.914050    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"]
Oct 30 19:34:34.942033 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.941979    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 19:34:34.942033 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.941994    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 19:34:34.942216 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.942159    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 19:34:34.942254 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.942231    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-klkgs\""
Oct 30 19:34:34.947234 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.947214    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 19:34:34.947334 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.947265    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 19:34:34.985847 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.985818    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2504d4c-a07f-4219-8c6b-948cc22da563-manifests\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.986005 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.985922    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2504d4c-a07f-4219-8c6b-948cc22da563-webhook-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.986005 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.985963    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmjdw\" (UniqueName: \"kubernetes.io/projected/c2504d4c-a07f-4219-8c6b-948cc22da563-kube-api-access-hmjdw\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.986113 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.986092    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2504d4c-a07f-4219-8c6b-948cc22da563-metrics-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.986162 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.986135    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2504d4c-a07f-4219-8c6b-948cc22da563-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.986162 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.986154    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2504d4c-a07f-4219-8c6b-948cc22da563-tmp\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.986298 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.986263    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2504d4c-a07f-4219-8c6b-948cc22da563-manifests\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.988301 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.988273    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2504d4c-a07f-4219-8c6b-948cc22da563-tmp\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.988418 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.988388    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2504d4c-a07f-4219-8c6b-948cc22da563-webhook-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.988504 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.988481    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2504d4c-a07f-4219-8c6b-948cc22da563-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.988592 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.988572    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2504d4c-a07f-4219-8c6b-948cc22da563-metrics-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:34.998518 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:34.998499    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmjdw\" (UniqueName: \"kubernetes.io/projected/c2504d4c-a07f-4219-8c6b-948cc22da563-kube-api-access-hmjdw\") pod \"opendatahub-operator-controller-manager-85768f955d-bfs2n\" (UID: \"c2504d4c-a07f-4219-8c6b-948cc22da563\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:35.181166 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:35.181072    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:35.481858 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:35.481776    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"]
Oct 30 19:34:35.489567 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:34:35.489531    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2504d4c_a07f_4219_8c6b_948cc22da563.slice/crio-ac7e706b1457321e512b8789d59d679560683d76ccd984cfa0ccbd5efc41b831 WatchSource:0}: Error finding container ac7e706b1457321e512b8789d59d679560683d76ccd984cfa0ccbd5efc41b831: Status 404 returned error can't find the container with id ac7e706b1457321e512b8789d59d679560683d76ccd984cfa0ccbd5efc41b831
Oct 30 19:34:35.611184 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:35.611152    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n" event={"ID":"c2504d4c-a07f-4219-8c6b-948cc22da563","Type":"ContainerStarted","Data":"ac7e706b1457321e512b8789d59d679560683d76ccd984cfa0ccbd5efc41b831"}
Oct 30 19:34:38.626487 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:38.626456    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n" event={"ID":"c2504d4c-a07f-4219-8c6b-948cc22da563","Type":"ContainerStarted","Data":"fc634d7c825c05d1034bfc2dd81b231b5f36be520404819d7c8d41e00623e23e"}
Oct 30 19:34:39.632531 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:39.632498    4823 generic.go:356] "Generic (PLEG): container finished" podID="c2504d4c-a07f-4219-8c6b-948cc22da563" containerID="fc634d7c825c05d1034bfc2dd81b231b5f36be520404819d7c8d41e00623e23e" exitCode=0
Oct 30 19:34:39.632956 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:39.632548    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n" event={"ID":"c2504d4c-a07f-4219-8c6b-948cc22da563","Type":"ContainerDied","Data":"fc634d7c825c05d1034bfc2dd81b231b5f36be520404819d7c8d41e00623e23e"}
Oct 30 19:34:40.639050 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:40.639012    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n" event={"ID":"c2504d4c-a07f-4219-8c6b-948cc22da563","Type":"ContainerStarted","Data":"e611871f2c18e195290265b130ea5357db0ffc46897c30be82851d4a8d0c97fc"}
Oct 30 19:34:40.639464 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:40.639134    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:40.689004 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:40.688953    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n" podStartSLOduration=2.981554299 podStartE2EDuration="6.688937264s" totalImagesPullingTime="3.242338367s" totalInitContainerRuntime="465.044598ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:40.688768581 +0000 UTC m=+280.581382917" watchObservedRunningTime="2026-10-30 19:34:40.688937264 +0000 UTC m=+280.581551590"
Oct 30 19:34:51.644946 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:51.644912    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-bfs2n"
Oct 30 19:34:53.144064 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:53.144028    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-fxhzx"]
Oct 30 19:34:53.148012 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:53.147992    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fxhzx"
Oct 30 19:34:53.154969 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:53.154949    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 19:34:53.155402 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:53.155385    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-z4hdv\""
Oct 30 19:34:53.155477 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:53.155403    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 19:34:53.159502 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:53.159481    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-fxhzx"]
Oct 30 19:34:53.238250 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:53.238213    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzwjs\" (UniqueName: \"kubernetes.io/projected/4833c70b-80ee-494f-be73-0de577c66b69-kube-api-access-mzwjs\") pod \"limitador-operator-controller-manager-5778bb4449-fxhzx\" (UID: \"4833c70b-80ee-494f-be73-0de577c66b69\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fxhzx"
Oct 30 19:34:53.339408 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:53.339379    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzwjs\" (UniqueName: \"kubernetes.io/projected/4833c70b-80ee-494f-be73-0de577c66b69-kube-api-access-mzwjs\") pod \"limitador-operator-controller-manager-5778bb4449-fxhzx\" (UID: \"4833c70b-80ee-494f-be73-0de577c66b69\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fxhzx"
Oct 30 19:34:53.357390 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:53.357365    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzwjs\" (UniqueName: \"kubernetes.io/projected/4833c70b-80ee-494f-be73-0de577c66b69-kube-api-access-mzwjs\") pod \"limitador-operator-controller-manager-5778bb4449-fxhzx\" (UID: \"4833c70b-80ee-494f-be73-0de577c66b69\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fxhzx"
Oct 30 19:34:53.460162 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:53.460097    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fxhzx"
Oct 30 19:34:53.744542 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:53.744520    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-fxhzx"]
Oct 30 19:34:53.747138 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:34:53.747106    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4833c70b_80ee_494f_be73_0de577c66b69.slice/crio-07bdd5be634354235b7ddc5224ea93f1963a71dc7fc676b10491fbdfbb681007 WatchSource:0}: Error finding container 07bdd5be634354235b7ddc5224ea93f1963a71dc7fc676b10491fbdfbb681007: Status 404 returned error can't find the container with id 07bdd5be634354235b7ddc5224ea93f1963a71dc7fc676b10491fbdfbb681007
Oct 30 19:34:54.697211 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:54.697173    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fxhzx" event={"ID":"4833c70b-80ee-494f-be73-0de577c66b69","Type":"ContainerStarted","Data":"07bdd5be634354235b7ddc5224ea93f1963a71dc7fc676b10491fbdfbb681007"}
Oct 30 19:34:56.620145 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:56.620117    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h469j"]
Oct 30 19:34:56.622959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:56.622939    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h469j"
Oct 30 19:34:56.627689 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:56.627669    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:34:56.627816 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:56.627743    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-kp52p\""
Oct 30 19:34:56.627816 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:56.627749    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 19:34:56.639825 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:56.639804    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h469j"]
Oct 30 19:34:56.715146 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:56.715107    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fxhzx" event={"ID":"4833c70b-80ee-494f-be73-0de577c66b69","Type":"ContainerStarted","Data":"8d2719d17c942388b9196569764344fd4732f507668ac8bce7b1b59951698fed"}
Oct 30 19:34:56.715324 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:56.715236    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fxhzx"
Oct 30 19:34:56.740275 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:56.740227    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fxhzx" podStartSLOduration=0.925462719 podStartE2EDuration="3.740214241s" totalImagesPullingTime="2.814751522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:56.739612421 +0000 UTC m=+296.632226744" watchObservedRunningTime="2026-10-30 19:34:56.740214241 +0000 UTC m=+296.632828565"
Oct 30 19:34:56.774030 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:56.773989    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b360250-fb5f-46ad-8e7c-e9630f2c2cee-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h469j\" (UID: \"1b360250-fb5f-46ad-8e7c-e9630f2c2cee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h469j"
Oct 30 19:34:56.774252 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:56.774051    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmk68\" (UniqueName: \"kubernetes.io/projected/1b360250-fb5f-46ad-8e7c-e9630f2c2cee-kube-api-access-nmk68\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h469j\" (UID: \"1b360250-fb5f-46ad-8e7c-e9630f2c2cee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h469j"
Oct 30 19:34:56.875329 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:56.875298    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b360250-fb5f-46ad-8e7c-e9630f2c2cee-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h469j\" (UID: \"1b360250-fb5f-46ad-8e7c-e9630f2c2cee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h469j"
Oct 30 19:34:56.875509 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:56.875342    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmk68\" (UniqueName: \"kubernetes.io/projected/1b360250-fb5f-46ad-8e7c-e9630f2c2cee-kube-api-access-nmk68\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h469j\" (UID: \"1b360250-fb5f-46ad-8e7c-e9630f2c2cee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h469j"
Oct 30 19:34:56.875693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:56.875671    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b360250-fb5f-46ad-8e7c-e9630f2c2cee-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h469j\" (UID: \"1b360250-fb5f-46ad-8e7c-e9630f2c2cee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h469j"
Oct 30 19:34:56.892166 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:56.892138    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmk68\" (UniqueName: \"kubernetes.io/projected/1b360250-fb5f-46ad-8e7c-e9630f2c2cee-kube-api-access-nmk68\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h469j\" (UID: \"1b360250-fb5f-46ad-8e7c-e9630f2c2cee\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h469j"
Oct 30 19:34:56.933692 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:56.933665    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h469j"
Oct 30 19:34:57.223759 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:57.223704    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h469j"]
Oct 30 19:34:57.227032 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:34:57.226997    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b360250_fb5f_46ad_8e7c_e9630f2c2cee.slice/crio-67d4101cd9f07e979df84bf0027b931e9e19530df404ea496980cc48f9c08c69 WatchSource:0}: Error finding container 67d4101cd9f07e979df84bf0027b931e9e19530df404ea496980cc48f9c08c69: Status 404 returned error can't find the container with id 67d4101cd9f07e979df84bf0027b931e9e19530df404ea496980cc48f9c08c69
Oct 30 19:34:57.720797 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:34:57.720765    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h469j" event={"ID":"1b360250-fb5f-46ad-8e7c-e9630f2c2cee","Type":"ContainerStarted","Data":"67d4101cd9f07e979df84bf0027b931e9e19530df404ea496980cc48f9c08c69"}
Oct 30 19:35:00.296889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:00.296838    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 19:35:00.297305 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:00.297289    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 19:35:02.013738 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:35:02.013695    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:35:07.723293 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:07.723259    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fxhzx"
Oct 30 19:35:09.506619 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:09.506585    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-grls6"]
Oct 30 19:35:09.511573 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:09.511556    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-grls6"
Oct 30 19:35:09.519421 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:09.519397    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-f4946\""
Oct 30 19:35:09.521440 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:09.521414    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-grls6"]
Oct 30 19:35:09.579661 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:09.579629    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkz4z\" (UniqueName: \"kubernetes.io/projected/442a8c9e-848f-4a76-974c-b2d56c290340-kube-api-access-xkz4z\") pod \"authorino-operator-7b7bb4bc77-grls6\" (UID: \"442a8c9e-848f-4a76-974c-b2d56c290340\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-grls6"
Oct 30 19:35:09.681117 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:09.681078    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkz4z\" (UniqueName: \"kubernetes.io/projected/442a8c9e-848f-4a76-974c-b2d56c290340-kube-api-access-xkz4z\") pod \"authorino-operator-7b7bb4bc77-grls6\" (UID: \"442a8c9e-848f-4a76-974c-b2d56c290340\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-grls6"
Oct 30 19:35:09.714974 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:09.714942    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkz4z\" (UniqueName: \"kubernetes.io/projected/442a8c9e-848f-4a76-974c-b2d56c290340-kube-api-access-xkz4z\") pod \"authorino-operator-7b7bb4bc77-grls6\" (UID: \"442a8c9e-848f-4a76-974c-b2d56c290340\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-grls6"
Oct 30 19:35:09.824493 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:09.824417    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-grls6"
Oct 30 19:35:10.109108 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:10.109076    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-grls6"]
Oct 30 19:35:10.112676 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:35:10.112649    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442a8c9e_848f_4a76_974c_b2d56c290340.slice/crio-0708ee26deeefad48874abacd77e96d618bc6e631941d039b6f6baf3b79f5bdd WatchSource:0}: Error finding container 0708ee26deeefad48874abacd77e96d618bc6e631941d039b6f6baf3b79f5bdd: Status 404 returned error can't find the container with id 0708ee26deeefad48874abacd77e96d618bc6e631941d039b6f6baf3b79f5bdd
Oct 30 19:35:10.114820 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:10.114802    4823 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:35:10.779540 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:10.779495    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-grls6" event={"ID":"442a8c9e-848f-4a76-974c-b2d56c290340","Type":"ContainerStarted","Data":"0708ee26deeefad48874abacd77e96d618bc6e631941d039b6f6baf3b79f5bdd"}
Oct 30 19:35:11.154569 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.154524    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"]
Oct 30 19:35:11.159437 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.159402    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.178122 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.178095    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"]
Oct 30 19:35:11.186963 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.186941    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-rcf7r\""
Oct 30 19:35:11.298299 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.298264    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.298472 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.298320    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.298472 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.298390    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.298579 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.298527    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.298579 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.298556    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.298673 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.298585    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.298673 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.298620    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.298782 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.298742    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h65hx\" (UniqueName: \"kubernetes.io/projected/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-kube-api-access-h65hx\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.298821 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.298796    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.399934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.399882    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.400112 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.399957    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.400112 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.399986    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.400112 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.400054    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.400112 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.400077    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.400112 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.400105    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.400380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.400138    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.400380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.400177    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h65hx\" (UniqueName: \"kubernetes.io/projected/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-kube-api-access-h65hx\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.400380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.400238    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.400535 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.400429    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.400607 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.400580    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.400671 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.400656    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.400915 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.400854    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.401329 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.401304    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.402723 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.402699    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.403057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.403033    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.431342 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.431285    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h65hx\" (UniqueName: \"kubernetes.io/projected/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-kube-api-access-h65hx\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.432903 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.432857    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911ae73c-fba8-4976-8c3c-38fb8ec9d20b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5ncl8n\" (UID: \"911ae73c-fba8-4976-8c3c-38fb8ec9d20b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.482032 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.481996    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:11.780328 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.779969    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"]
Oct 30 19:35:11.783796 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:35:11.783763    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911ae73c_fba8_4976_8c3c_38fb8ec9d20b.slice/crio-48e4b21ae635b183a2716c9fcaa01abb8dca87b7f7cf19a3da8aa8f26b8066a5 WatchSource:0}: Error finding container 48e4b21ae635b183a2716c9fcaa01abb8dca87b7f7cf19a3da8aa8f26b8066a5: Status 404 returned error can't find the container with id 48e4b21ae635b183a2716c9fcaa01abb8dca87b7f7cf19a3da8aa8f26b8066a5
Oct 30 19:35:11.786439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.786407    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:35:11.786537 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.786472    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:35:11.786537 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:11.786501    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:35:12.791522 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:12.791484    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h469j" event={"ID":"1b360250-fb5f-46ad-8e7c-e9630f2c2cee","Type":"ContainerStarted","Data":"0651263ae077a3ca63684d564bcbb490fa08e1135194b0b4d4ba6c4101fe39c6"}
Oct 30 19:35:12.793169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:12.793144    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-grls6" event={"ID":"442a8c9e-848f-4a76-974c-b2d56c290340","Type":"ContainerStarted","Data":"c4eef114fee9c0dac54ae0cec5b870a3c067022391dad962f28015dc380751fb"}
Oct 30 19:35:12.793309 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:12.793190    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-grls6"
Oct 30 19:35:12.794898 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:12.794850    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n" event={"ID":"911ae73c-fba8-4976-8c3c-38fb8ec9d20b","Type":"ContainerStarted","Data":"316c764ceb4aa992197811bfc98858f2feedfb59272d9f3d3e24e537449711d4"}
Oct 30 19:35:12.794997 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:12.794917    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n" event={"ID":"911ae73c-fba8-4976-8c3c-38fb8ec9d20b","Type":"ContainerStarted","Data":"48e4b21ae635b183a2716c9fcaa01abb8dca87b7f7cf19a3da8aa8f26b8066a5"}
Oct 30 19:35:12.820778 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:12.820732    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h469j" podStartSLOduration=1.6749583000000001 podStartE2EDuration="16.820718525s" totalImagesPullingTime="15.145760225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:12.818935505 +0000 UTC m=+312.711549826" watchObservedRunningTime="2026-10-30 19:35:12.820718525 +0000 UTC m=+312.713332846"
Oct 30 19:35:12.850110 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:12.850058    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n" podStartSLOduration=1.85004488 podStartE2EDuration="1.85004488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:12.849808287 +0000 UTC m=+312.742422613" watchObservedRunningTime="2026-10-30 19:35:12.85004488 +0000 UTC m=+312.742659204"
Oct 30 19:35:12.871968 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:12.871919    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-grls6" podStartSLOduration=1.38494786 podStartE2EDuration="3.871904489s" totalImagesPullingTime="2.486956629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:12.870961789 +0000 UTC m=+312.763576125" watchObservedRunningTime="2026-10-30 19:35:12.871904489 +0000 UTC m=+312.764518813"
Oct 30 19:35:13.482665 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:13.482629    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:13.487453 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:13.487429    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:13.800222 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:13.800148    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:13.800794 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:13.800779    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5ncl8n"
Oct 30 19:35:15.339542 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.339510    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"]
Oct 30 19:35:15.343091 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.343073    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"
Oct 30 19:35:15.352823 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.352800    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-mtw7d\""
Oct 30 19:35:15.368546 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.368526    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"]
Oct 30 19:35:15.386709 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.386683    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 19:35:15.386816 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.386729    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 19:35:15.389061 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.389042    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 19:35:15.440945 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.440906    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/671e4edd-8d1a-4f5d-8375-1bba1686d004-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-t5mbc\" (UID: \"671e4edd-8d1a-4f5d-8375-1bba1686d004\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"
Oct 30 19:35:15.441134 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.440953    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd9ls\" (UniqueName: \"kubernetes.io/projected/671e4edd-8d1a-4f5d-8375-1bba1686d004-kube-api-access-hd9ls\") pod \"kserve-module-controller-manager-7566cf5fc8-t5mbc\" (UID: \"671e4edd-8d1a-4f5d-8375-1bba1686d004\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"
Oct 30 19:35:15.441134 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.441049    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/671e4edd-8d1a-4f5d-8375-1bba1686d004-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-t5mbc\" (UID: \"671e4edd-8d1a-4f5d-8375-1bba1686d004\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"
Oct 30 19:35:15.441134 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.441120    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/671e4edd-8d1a-4f5d-8375-1bba1686d004-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-t5mbc\" (UID: \"671e4edd-8d1a-4f5d-8375-1bba1686d004\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"
Oct 30 19:35:15.542108 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.542074    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/671e4edd-8d1a-4f5d-8375-1bba1686d004-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-t5mbc\" (UID: \"671e4edd-8d1a-4f5d-8375-1bba1686d004\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"
Oct 30 19:35:15.542304 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.542121    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd9ls\" (UniqueName: \"kubernetes.io/projected/671e4edd-8d1a-4f5d-8375-1bba1686d004-kube-api-access-hd9ls\") pod \"kserve-module-controller-manager-7566cf5fc8-t5mbc\" (UID: \"671e4edd-8d1a-4f5d-8375-1bba1686d004\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"
Oct 30 19:35:15.542304 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.542154    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/671e4edd-8d1a-4f5d-8375-1bba1686d004-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-t5mbc\" (UID: \"671e4edd-8d1a-4f5d-8375-1bba1686d004\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"
Oct 30 19:35:15.542304 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.542171    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/671e4edd-8d1a-4f5d-8375-1bba1686d004-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-t5mbc\" (UID: \"671e4edd-8d1a-4f5d-8375-1bba1686d004\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"
Oct 30 19:35:15.542564 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.542538    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/671e4edd-8d1a-4f5d-8375-1bba1686d004-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-t5mbc\" (UID: \"671e4edd-8d1a-4f5d-8375-1bba1686d004\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"
Oct 30 19:35:15.542625 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.542596    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/671e4edd-8d1a-4f5d-8375-1bba1686d004-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-t5mbc\" (UID: \"671e4edd-8d1a-4f5d-8375-1bba1686d004\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"
Oct 30 19:35:15.542959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.542943    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/671e4edd-8d1a-4f5d-8375-1bba1686d004-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-t5mbc\" (UID: \"671e4edd-8d1a-4f5d-8375-1bba1686d004\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"
Oct 30 19:35:15.556531 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.556496    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd9ls\" (UniqueName: \"kubernetes.io/projected/671e4edd-8d1a-4f5d-8375-1bba1686d004-kube-api-access-hd9ls\") pod \"kserve-module-controller-manager-7566cf5fc8-t5mbc\" (UID: \"671e4edd-8d1a-4f5d-8375-1bba1686d004\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"
Oct 30 19:35:15.655357 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.655319    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"
Oct 30 19:35:15.934059 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:15.933973    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"]
Oct 30 19:35:15.936780 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:35:15.936746    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671e4edd_8d1a_4f5d_8375_1bba1686d004.slice/crio-198bb5d64abf3ab42ed70b167385a9d730ccf882ed4d392b38bb4e9e3445663c WatchSource:0}: Error finding container 198bb5d64abf3ab42ed70b167385a9d730ccf882ed4d392b38bb4e9e3445663c: Status 404 returned error can't find the container with id 198bb5d64abf3ab42ed70b167385a9d730ccf882ed4d392b38bb4e9e3445663c
Oct 30 19:35:16.819004 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:16.818958    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc" event={"ID":"671e4edd-8d1a-4f5d-8375-1bba1686d004","Type":"ContainerStarted","Data":"198bb5d64abf3ab42ed70b167385a9d730ccf882ed4d392b38bb4e9e3445663c"}
Oct 30 19:35:18.829354 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:18.829320    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc" event={"ID":"671e4edd-8d1a-4f5d-8375-1bba1686d004","Type":"ContainerStarted","Data":"f4230df73a1a7da9166136fe7d5b4404b5f34dc5b68cf0dc717ac321053cf79d"}
Oct 30 19:35:18.829756 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:18.829395    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"
Oct 30 19:35:18.877853 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:18.877785    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc" podStartSLOduration=1.5645001779999999 podStartE2EDuration="3.87777055s" totalImagesPullingTime="2.313270372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:18.876345131 +0000 UTC m=+318.768959455" watchObservedRunningTime="2026-10-30 19:35:18.87777055 +0000 UTC m=+318.770384874"
Oct 30 19:35:23.802009 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:23.801970    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-grls6"
Oct 30 19:35:32.013961 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:35:32.013921    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:35:39.835352 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:35:39.835320    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-t5mbc"
Oct 30 19:36:02.013198 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:36:02.013157    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:36:10.299842 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.299807    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-kn9dm"]
Oct 30 19:36:10.312683 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.312656    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-kn9dm"
Oct 30 19:36:10.315218 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.315188    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-kn9dm"]
Oct 30 19:36:10.321538 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.321510    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 19:36:10.323811 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.323792    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-s8xls\""
Oct 30 19:36:10.342194 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.342163    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vql5z"]
Oct 30 19:36:10.345663 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.345641    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-vql5z"
Oct 30 19:36:10.367121 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.367090    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vql5z"]
Oct 30 19:36:10.375085 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.375057    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-ntzq9"]
Oct 30 19:36:10.378461 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.378443    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ntzq9"
Oct 30 19:36:10.396684 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.396640    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ntzq9"]
Oct 30 19:36:10.398823 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.398799    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bkdcs\""
Oct 30 19:36:10.398982 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.398803    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 19:36:10.398982 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.398878    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-647sx\""
Oct 30 19:36:10.398982 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.398916    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 19:36:10.399122 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.399074    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 19:36:10.424410 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.424376    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2e03817-be34-4666-befb-3a6cd283c45c-cert\") pod \"kserve-controller-manager-776cc66d47-kn9dm\" (UID: \"f2e03817-be34-4666-befb-3a6cd283c45c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kn9dm"
Oct 30 19:36:10.424581 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.424416    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vwzp\" (UniqueName: \"kubernetes.io/projected/f2e03817-be34-4666-befb-3a6cd283c45c-kube-api-access-2vwzp\") pod \"kserve-controller-manager-776cc66d47-kn9dm\" (UID: \"f2e03817-be34-4666-befb-3a6cd283c45c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kn9dm"
Oct 30 19:36:10.525710 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.525677    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aec94275-86c6-43e2-88d5-ceeed0756ebd-metrics-certs\") pod \"odh-model-controller-769875db56-ntzq9\" (UID: \"aec94275-86c6-43e2-88d5-ceeed0756ebd\") " pod="opendatahub/odh-model-controller-769875db56-ntzq9"
Oct 30 19:36:10.525984 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.525719    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beba841c-4b97-40ec-a959-f9130317dc4f-tls-certs\") pod \"model-serving-api-75bd89fdfc-vql5z\" (UID: \"beba841c-4b97-40ec-a959-f9130317dc4f\") " pod="opendatahub/model-serving-api-75bd89fdfc-vql5z"
Oct 30 19:36:10.525984 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.525740    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9p9z\" (UniqueName: \"kubernetes.io/projected/aec94275-86c6-43e2-88d5-ceeed0756ebd-kube-api-access-f9p9z\") pod \"odh-model-controller-769875db56-ntzq9\" (UID: \"aec94275-86c6-43e2-88d5-ceeed0756ebd\") " pod="opendatahub/odh-model-controller-769875db56-ntzq9"
Oct 30 19:36:10.525984 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.525816    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh8qt\" (UniqueName: \"kubernetes.io/projected/beba841c-4b97-40ec-a959-f9130317dc4f-kube-api-access-qh8qt\") pod \"model-serving-api-75bd89fdfc-vql5z\" (UID: \"beba841c-4b97-40ec-a959-f9130317dc4f\") " pod="opendatahub/model-serving-api-75bd89fdfc-vql5z"
Oct 30 19:36:10.525984 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.525857    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aec94275-86c6-43e2-88d5-ceeed0756ebd-cert\") pod \"odh-model-controller-769875db56-ntzq9\" (UID: \"aec94275-86c6-43e2-88d5-ceeed0756ebd\") " pod="opendatahub/odh-model-controller-769875db56-ntzq9"
Oct 30 19:36:10.526221 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.526036    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2e03817-be34-4666-befb-3a6cd283c45c-cert\") pod \"kserve-controller-manager-776cc66d47-kn9dm\" (UID: \"f2e03817-be34-4666-befb-3a6cd283c45c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kn9dm"
Oct 30 19:36:10.526221 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.526086    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vwzp\" (UniqueName: \"kubernetes.io/projected/f2e03817-be34-4666-befb-3a6cd283c45c-kube-api-access-2vwzp\") pod \"kserve-controller-manager-776cc66d47-kn9dm\" (UID: \"f2e03817-be34-4666-befb-3a6cd283c45c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kn9dm"
Oct 30 19:36:10.528633 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.528605    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2e03817-be34-4666-befb-3a6cd283c45c-cert\") pod \"kserve-controller-manager-776cc66d47-kn9dm\" (UID: \"f2e03817-be34-4666-befb-3a6cd283c45c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kn9dm"
Oct 30 19:36:10.540460 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.540434    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vwzp\" (UniqueName: \"kubernetes.io/projected/f2e03817-be34-4666-befb-3a6cd283c45c-kube-api-access-2vwzp\") pod \"kserve-controller-manager-776cc66d47-kn9dm\" (UID: \"f2e03817-be34-4666-befb-3a6cd283c45c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-kn9dm"
Oct 30 19:36:10.627242 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.627199    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aec94275-86c6-43e2-88d5-ceeed0756ebd-metrics-certs\") pod \"odh-model-controller-769875db56-ntzq9\" (UID: \"aec94275-86c6-43e2-88d5-ceeed0756ebd\") " pod="opendatahub/odh-model-controller-769875db56-ntzq9"
Oct 30 19:36:10.627416 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.627257    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beba841c-4b97-40ec-a959-f9130317dc4f-tls-certs\") pod \"model-serving-api-75bd89fdfc-vql5z\" (UID: \"beba841c-4b97-40ec-a959-f9130317dc4f\") " pod="opendatahub/model-serving-api-75bd89fdfc-vql5z"
Oct 30 19:36:10.627416 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.627285    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9p9z\" (UniqueName: \"kubernetes.io/projected/aec94275-86c6-43e2-88d5-ceeed0756ebd-kube-api-access-f9p9z\") pod \"odh-model-controller-769875db56-ntzq9\" (UID: \"aec94275-86c6-43e2-88d5-ceeed0756ebd\") " pod="opendatahub/odh-model-controller-769875db56-ntzq9"
Oct 30 19:36:10.627416 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.627314    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh8qt\" (UniqueName: \"kubernetes.io/projected/beba841c-4b97-40ec-a959-f9130317dc4f-kube-api-access-qh8qt\") pod \"model-serving-api-75bd89fdfc-vql5z\" (UID: \"beba841c-4b97-40ec-a959-f9130317dc4f\") " pod="opendatahub/model-serving-api-75bd89fdfc-vql5z"
Oct 30 19:36:10.627416 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.627338    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aec94275-86c6-43e2-88d5-ceeed0756ebd-cert\") pod \"odh-model-controller-769875db56-ntzq9\" (UID: \"aec94275-86c6-43e2-88d5-ceeed0756ebd\") " pod="opendatahub/odh-model-controller-769875db56-ntzq9"
Oct 30 19:36:10.629936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.629912    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aec94275-86c6-43e2-88d5-ceeed0756ebd-metrics-certs\") pod \"odh-model-controller-769875db56-ntzq9\" (UID: \"aec94275-86c6-43e2-88d5-ceeed0756ebd\") " pod="opendatahub/odh-model-controller-769875db56-ntzq9"
Oct 30 19:36:10.629936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.629933    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aec94275-86c6-43e2-88d5-ceeed0756ebd-cert\") pod \"odh-model-controller-769875db56-ntzq9\" (UID: \"aec94275-86c6-43e2-88d5-ceeed0756ebd\") " pod="opendatahub/odh-model-controller-769875db56-ntzq9"
Oct 30 19:36:10.630100 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.630026    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beba841c-4b97-40ec-a959-f9130317dc4f-tls-certs\") pod \"model-serving-api-75bd89fdfc-vql5z\" (UID: \"beba841c-4b97-40ec-a959-f9130317dc4f\") " pod="opendatahub/model-serving-api-75bd89fdfc-vql5z"
Oct 30 19:36:10.630100 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.630059    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-kn9dm"
Oct 30 19:36:10.647311 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.647290    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9p9z\" (UniqueName: \"kubernetes.io/projected/aec94275-86c6-43e2-88d5-ceeed0756ebd-kube-api-access-f9p9z\") pod \"odh-model-controller-769875db56-ntzq9\" (UID: \"aec94275-86c6-43e2-88d5-ceeed0756ebd\") " pod="opendatahub/odh-model-controller-769875db56-ntzq9"
Oct 30 19:36:10.647623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.647604    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh8qt\" (UniqueName: \"kubernetes.io/projected/beba841c-4b97-40ec-a959-f9130317dc4f-kube-api-access-qh8qt\") pod \"model-serving-api-75bd89fdfc-vql5z\" (UID: \"beba841c-4b97-40ec-a959-f9130317dc4f\") " pod="opendatahub/model-serving-api-75bd89fdfc-vql5z"
Oct 30 19:36:10.662432 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.662403    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-vql5z"
Oct 30 19:36:10.695243 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.695215    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ntzq9"
Oct 30 19:36:10.913793 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.913724    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-kn9dm"]
Oct 30 19:36:10.917074 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:36:10.917042    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e03817_be34_4666_befb_3a6cd283c45c.slice/crio-8d604bf7fda4bdf06a12c04a137232fba0e9a9058a7ba91134e3ef0bc735b945 WatchSource:0}: Error finding container 8d604bf7fda4bdf06a12c04a137232fba0e9a9058a7ba91134e3ef0bc735b945: Status 404 returned error can't find the container with id 8d604bf7fda4bdf06a12c04a137232fba0e9a9058a7ba91134e3ef0bc735b945
Oct 30 19:36:10.952883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:10.952828    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vql5z"]
Oct 30 19:36:10.955831 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:36:10.955801    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeba841c_4b97_40ec_a959_f9130317dc4f.slice/crio-db460425ed924d9837db45abab77bdc7aee2b1385f034f43a3243685b9f9e338 WatchSource:0}: Error finding container db460425ed924d9837db45abab77bdc7aee2b1385f034f43a3243685b9f9e338: Status 404 returned error can't find the container with id db460425ed924d9837db45abab77bdc7aee2b1385f034f43a3243685b9f9e338
Oct 30 19:36:11.014113 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:11.014085    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ntzq9"]
Oct 30 19:36:11.016355 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:36:11.016326    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec94275_86c6_43e2_88d5_ceeed0756ebd.slice/crio-57058f45368a12c676b28b818a8b003b574b219364621366e99bcb10a683c719 WatchSource:0}: Error finding container 57058f45368a12c676b28b818a8b003b574b219364621366e99bcb10a683c719: Status 404 returned error can't find the container with id 57058f45368a12c676b28b818a8b003b574b219364621366e99bcb10a683c719
Oct 30 19:36:11.081623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:11.081588    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-vql5z" event={"ID":"beba841c-4b97-40ec-a959-f9130317dc4f","Type":"ContainerStarted","Data":"db460425ed924d9837db45abab77bdc7aee2b1385f034f43a3243685b9f9e338"}
Oct 30 19:36:11.082768 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:11.082735    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ntzq9" event={"ID":"aec94275-86c6-43e2-88d5-ceeed0756ebd","Type":"ContainerStarted","Data":"57058f45368a12c676b28b818a8b003b574b219364621366e99bcb10a683c719"}
Oct 30 19:36:11.083876 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:11.083839    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-kn9dm" event={"ID":"f2e03817-be34-4666-befb-3a6cd283c45c","Type":"ContainerStarted","Data":"8d604bf7fda4bdf06a12c04a137232fba0e9a9058a7ba91134e3ef0bc735b945"}
Oct 30 19:36:14.108317 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:14.108274    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-kn9dm" event={"ID":"f2e03817-be34-4666-befb-3a6cd283c45c","Type":"ContainerStarted","Data":"5272ec35308152e482f661537c0c779529925f02411d7cdf94e0944da3792756"}
Oct 30 19:36:14.108797 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:14.108355    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-kn9dm"
Oct 30 19:36:14.109971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:14.109948    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-vql5z" event={"ID":"beba841c-4b97-40ec-a959-f9130317dc4f","Type":"ContainerStarted","Data":"76791c930e0bf7db246c3f3444e17c82d07574cd6ab2058fed533c736fe21e9d"}
Oct 30 19:36:14.110114 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:14.110094    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-vql5z"
Oct 30 19:36:14.128027 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:14.127982    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-kn9dm" podStartSLOduration=1.400369531 podStartE2EDuration="4.127968693s" totalImagesPullingTime="2.727599162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:14.125878751 +0000 UTC m=+374.018493076" watchObservedRunningTime="2026-10-30 19:36:14.127968693 +0000 UTC m=+374.020583016"
Oct 30 19:36:14.145296 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:14.145248    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-vql5z" podStartSLOduration=1.808806649 podStartE2EDuration="4.14523317s" totalImagesPullingTime="2.336426521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:14.144148083 +0000 UTC m=+374.036762406" watchObservedRunningTime="2026-10-30 19:36:14.14523317 +0000 UTC m=+374.037847495"
Oct 30 19:36:15.116162 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:15.116127    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ntzq9" event={"ID":"aec94275-86c6-43e2-88d5-ceeed0756ebd","Type":"ContainerStarted","Data":"0d8f691871d3ba8c0eef0813249391f2a968ab8a3393c6e3f4f3985fcf5b3938"}
Oct 30 19:36:15.116652 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:15.116482    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-ntzq9"
Oct 30 19:36:15.135733 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:15.135683    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-ntzq9" podStartSLOduration=1.980768239 podStartE2EDuration="5.135669686s" totalImagesPullingTime="3.154901447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:15.134158595 +0000 UTC m=+375.026772911" watchObservedRunningTime="2026-10-30 19:36:15.135669686 +0000 UTC m=+375.028284009"
Oct 30 19:36:25.120877 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:25.120835    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-vql5z"
Oct 30 19:36:26.124187 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:26.124153    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-ntzq9"
Oct 30 19:36:32.013857 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:36:32.013814    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:36:45.122219 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:36:45.122146    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-kn9dm"
Oct 30 19:37:02.013036 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:37:02.012997    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:37:32.013725 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:37:32.013686    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:38:02.012057 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:38:02.012014    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:38:32.013625 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:38:32.013539    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:39:02.013222 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:39:02.013182    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:39:32.013276 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:39:32.013175    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:40:00.327305 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:40:00.327274    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 19:40:00.329010 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:40:00.328982    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 19:40:02.012741 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:40:02.012708    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:40:32.012795 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:40:32.012749    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:41:02.013998 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:41:02.013959    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:41:32.014374 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:41:32.014289    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:42:02.014443 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:42:02.014405    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:42:32.013962 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:42:32.013917    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:43:02.013597 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:43:02.013555    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:43:32.013322 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:43:32.013279    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:44:02.016516 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:44:02.014111    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:44:32.014651 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:44:32.014539    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:45:00.373040 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:00.372999    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 19:45:00.374329 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:00.374303    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 19:45:02.014075 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:45:02.014033    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:45:32.015911 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:45:32.015847    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:45:50.396954 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.396537    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt"]
Oct 30 19:45:50.400457 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.400429    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt"
Oct 30 19:45:50.408459 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.408421    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 19:45:50.408667 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.408559    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-q2knc\""
Oct 30 19:45:50.411325 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.411298    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt"]
Oct 30 19:45:50.411912 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.411856    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 19:45:50.533487 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.533453    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/712ee475-078f-488e-aabb-35489b8bda90-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vkgjt\" (UID: \"712ee475-078f-488e-aabb-35489b8bda90\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt"
Oct 30 19:45:50.533660 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.533523    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw475\" (UniqueName: \"kubernetes.io/projected/712ee475-078f-488e-aabb-35489b8bda90-kube-api-access-qw475\") pod \"kuadrant-console-plugin-66c4fb575d-vkgjt\" (UID: \"712ee475-078f-488e-aabb-35489b8bda90\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt"
Oct 30 19:45:50.533660 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.533566    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/712ee475-078f-488e-aabb-35489b8bda90-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vkgjt\" (UID: \"712ee475-078f-488e-aabb-35489b8bda90\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt"
Oct 30 19:45:50.634404 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.634361    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw475\" (UniqueName: \"kubernetes.io/projected/712ee475-078f-488e-aabb-35489b8bda90-kube-api-access-qw475\") pod \"kuadrant-console-plugin-66c4fb575d-vkgjt\" (UID: \"712ee475-078f-488e-aabb-35489b8bda90\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt"
Oct 30 19:45:50.634659 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.634447    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/712ee475-078f-488e-aabb-35489b8bda90-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vkgjt\" (UID: \"712ee475-078f-488e-aabb-35489b8bda90\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt"
Oct 30 19:45:50.634659 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.634587    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/712ee475-078f-488e-aabb-35489b8bda90-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vkgjt\" (UID: \"712ee475-078f-488e-aabb-35489b8bda90\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt"
Oct 30 19:45:50.635511 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.635482    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/712ee475-078f-488e-aabb-35489b8bda90-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vkgjt\" (UID: \"712ee475-078f-488e-aabb-35489b8bda90\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt"
Oct 30 19:45:50.650302 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.650216    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw475\" (UniqueName: \"kubernetes.io/projected/712ee475-078f-488e-aabb-35489b8bda90-kube-api-access-qw475\") pod \"kuadrant-console-plugin-66c4fb575d-vkgjt\" (UID: \"712ee475-078f-488e-aabb-35489b8bda90\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt"
Oct 30 19:45:50.654068 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.653773    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/712ee475-078f-488e-aabb-35489b8bda90-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vkgjt\" (UID: \"712ee475-078f-488e-aabb-35489b8bda90\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt"
Oct 30 19:45:50.728680 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.728642    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:45:50.732436 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.732416    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"
Oct 30 19:45:50.732776 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.732744    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt"
Oct 30 19:45:50.742675 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.742643    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:45:50.755667 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.755621    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 19:45:50.781727 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.781685    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:45:50.836350 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.836305    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfmpk\" (UniqueName: \"kubernetes.io/projected/1b499879-f7cb-4f24-95f4-083df601dd40-kube-api-access-jfmpk\") pod \"limitador-limitador-7c9b87d574-gdgrn\" (UID: \"1b499879-f7cb-4f24-95f4-083df601dd40\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"
Oct 30 19:45:50.836531 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.836424    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1b499879-f7cb-4f24-95f4-083df601dd40-config-file\") pod \"limitador-limitador-7c9b87d574-gdgrn\" (UID: \"1b499879-f7cb-4f24-95f4-083df601dd40\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"
Oct 30 19:45:50.937814 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.937726    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1b499879-f7cb-4f24-95f4-083df601dd40-config-file\") pod \"limitador-limitador-7c9b87d574-gdgrn\" (UID: \"1b499879-f7cb-4f24-95f4-083df601dd40\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"
Oct 30 19:45:50.937988 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.937905    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfmpk\" (UniqueName: \"kubernetes.io/projected/1b499879-f7cb-4f24-95f4-083df601dd40-kube-api-access-jfmpk\") pod \"limitador-limitador-7c9b87d574-gdgrn\" (UID: \"1b499879-f7cb-4f24-95f4-083df601dd40\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"
Oct 30 19:45:50.938431 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.938413    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1b499879-f7cb-4f24-95f4-083df601dd40-config-file\") pod \"limitador-limitador-7c9b87d574-gdgrn\" (UID: \"1b499879-f7cb-4f24-95f4-083df601dd40\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"
Oct 30 19:45:50.950320 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:50.950288    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfmpk\" (UniqueName: \"kubernetes.io/projected/1b499879-f7cb-4f24-95f4-083df601dd40-kube-api-access-jfmpk\") pod \"limitador-limitador-7c9b87d574-gdgrn\" (UID: \"1b499879-f7cb-4f24-95f4-083df601dd40\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"
Oct 30 19:45:51.018894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:51.018844    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt"]
Oct 30 19:45:51.022081 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:45:51.022045    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod712ee475_078f_488e_aabb_35489b8bda90.slice/crio-716c4b9f5ce24a984d36f9dc3789d5b8eb37636fafa40ba156d174a60db7b598 WatchSource:0}: Error finding container 716c4b9f5ce24a984d36f9dc3789d5b8eb37636fafa40ba156d174a60db7b598: Status 404 returned error can't find the container with id 716c4b9f5ce24a984d36f9dc3789d5b8eb37636fafa40ba156d174a60db7b598
Oct 30 19:45:51.023386 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:51.023365    4823 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:45:51.068511 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:51.068479    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"
Oct 30 19:45:51.354481 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:51.354404    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:45:51.357765 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:45:51.357734    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b499879_f7cb_4f24_95f4_083df601dd40.slice/crio-31fa0df976843c61a85c0c6fdaf80db3367879b25bb710828c1f5a003670965a WatchSource:0}: Error finding container 31fa0df976843c61a85c0c6fdaf80db3367879b25bb710828c1f5a003670965a: Status 404 returned error can't find the container with id 31fa0df976843c61a85c0c6fdaf80db3367879b25bb710828c1f5a003670965a
Oct 30 19:45:51.834717 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:51.834678    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-gdgrn" event={"ID":"1b499879-f7cb-4f24-95f4-083df601dd40","Type":"ContainerStarted","Data":"31fa0df976843c61a85c0c6fdaf80db3367879b25bb710828c1f5a003670965a"}
Oct 30 19:45:51.835856 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:51.835828    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt" event={"ID":"712ee475-078f-488e-aabb-35489b8bda90","Type":"ContainerStarted","Data":"716c4b9f5ce24a984d36f9dc3789d5b8eb37636fafa40ba156d174a60db7b598"}
Oct 30 19:45:52.844758 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:52.844717    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-gdgrn" event={"ID":"1b499879-f7cb-4f24-95f4-083df601dd40","Type":"ContainerStarted","Data":"77d3edcc943bd0a4e5bf465f68800c76f537505c9ce50cbb32ba560fec6a1d39"}
Oct 30 19:45:52.845250 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:52.844885    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"
Oct 30 19:45:52.865085 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:52.865022    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-gdgrn" podStartSLOduration=1.5768449850000001 podStartE2EDuration="2.865002008s" totalImagesPullingTime="1.288157023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:52.864718582 +0000 UTC m=+952.757332907" watchObservedRunningTime="2026-10-30 19:45:52.865002008 +0000 UTC m=+952.757616335"
Oct 30 19:45:56.869285 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:56.869248    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt" event={"ID":"712ee475-078f-488e-aabb-35489b8bda90","Type":"ContainerStarted","Data":"e43926cae272a0aba00e70cddb4ff59cd0abbf32cf10d4dd806dc55a45cb8730"}
Oct 30 19:45:56.887920 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:45:56.887834    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vkgjt" podStartSLOduration=1.719210934 podStartE2EDuration="6.88781527s" totalImagesPullingTime="5.168604336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:56.887556458 +0000 UTC m=+956.780170783" watchObservedRunningTime="2026-10-30 19:45:56.88781527 +0000 UTC m=+956.780429597"
Oct 30 19:46:02.014613 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:46:02.014572    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:46:03.851761 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:46:03.851728    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"
Oct 30 19:46:32.014455 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:46:32.014416    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:47:02.012534 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:47:02.012493    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:47:06.097351 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:06.097317    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:47:15.530377 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:15.530339    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-6p5n6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": read tcp 10.132.0.2:43520->10.132.0.32:15021: read: connection reset by peer" start-of-body=
Oct 30 19:47:15.530816 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:15.530417    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6" podUID="892f7495-a9e2-489d-92c5-12f35e1e65f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": read tcp 10.132.0.2:43520->10.132.0.32:15021: read: connection reset by peer"
Oct 30 19:47:16.255438 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:16.255403    4823 generic.go:356] "Generic (PLEG): container finished" podID="892f7495-a9e2-489d-92c5-12f35e1e65f3" containerID="0945026cbe09b6d844248ef4720c838b050a20ad9e4917fb079c1d1c5b6f12e8" exitCode=0
Oct 30 19:47:16.255606 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:16.255479    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6" event={"ID":"892f7495-a9e2-489d-92c5-12f35e1e65f3","Type":"ContainerDied","Data":"0945026cbe09b6d844248ef4720c838b050a20ad9e4917fb079c1d1c5b6f12e8"}
Oct 30 19:47:16.255847 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:16.255820    4823 scope.go:122] "RemoveContainer" containerID="0945026cbe09b6d844248ef4720c838b050a20ad9e4917fb079c1d1c5b6f12e8"
Oct 30 19:47:16.256955 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:16.256928    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:47:16.257016 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:16.256995    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:47:16.257052 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:16.257027    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:47:17.279684 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:17.279649    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6" event={"ID":"892f7495-a9e2-489d-92c5-12f35e1e65f3","Type":"ContainerStarted","Data":"3e43558ce2d5fcab19826a43fe451dc4df5d0e8c6818dc5f3d0c598b6b8eecd3"}
Oct 30 19:47:17.301107 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:17.301075    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:47:17.301263 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:17.301204    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-6p5n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Oct 30 19:47:17.301263 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:17.301244    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6" podUID="892f7495-a9e2-489d-92c5-12f35e1e65f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Oct 30 19:47:18.300961 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:18.300928    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-6p5n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Oct 30 19:47:18.301360 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:18.300976    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6" podUID="892f7495-a9e2-489d-92c5-12f35e1e65f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Oct 30 19:47:19.301133 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:19.301100    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-6p5n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Oct 30 19:47:19.301551 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:19.301155    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6" podUID="892f7495-a9e2-489d-92c5-12f35e1e65f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Oct 30 19:47:20.301390 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:20.301353    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-6p5n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Oct 30 19:47:20.301784 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:20.301419    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6" podUID="892f7495-a9e2-489d-92c5-12f35e1e65f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Oct 30 19:47:21.301373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:21.301338    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-6p5n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Oct 30 19:47:21.301551 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:21.301386    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6" podUID="892f7495-a9e2-489d-92c5-12f35e1e65f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Oct 30 19:47:22.303829 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:22.303801    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:47:22.304284 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:22.304062    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:47:22.304680 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:22.304660    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6"
Oct 30 19:47:32.013212 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:47:32.013171    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:47:41.059821 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.059789    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"]
Oct 30 19:47:41.062538 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.062515    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.091203 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.091172    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"]
Oct 30 19:47:41.165147 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.165115    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7d2k\" (UniqueName: \"kubernetes.io/projected/3eb29b61-5760-4a85-8573-a3205e66e833-kube-api-access-p7d2k\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.165147 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.165151    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb29b61-5760-4a85-8573-a3205e66e833-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.165354 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.165173    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb29b61-5760-4a85-8573-a3205e66e833-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.165354 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.165318    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.165467 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.165396    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.165467 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.165421    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.165467 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.165455    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb29b61-5760-4a85-8573-a3205e66e833-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.165563 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.165479    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.165563 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.165495    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.266722 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.266687    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.266935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.266738    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.266935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.266761    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.266935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.266786    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb29b61-5760-4a85-8573-a3205e66e833-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.266935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.266821    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.266935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.266847    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.266935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.266899    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7d2k\" (UniqueName: \"kubernetes.io/projected/3eb29b61-5760-4a85-8573-a3205e66e833-kube-api-access-p7d2k\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.266935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.266925    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb29b61-5760-4a85-8573-a3205e66e833-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.267278 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.266954    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb29b61-5760-4a85-8573-a3205e66e833-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.267278 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.267236    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.267384 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.267322    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.267447 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.267433    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.267598 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.267576    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.267772 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.267743    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb29b61-5760-4a85-8573-a3205e66e833-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.269068 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.269043    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.269304 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.269288    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb29b61-5760-4a85-8573-a3205e66e833-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.279638 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.279616    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb29b61-5760-4a85-8573-a3205e66e833-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.279916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.279837    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7d2k\" (UniqueName: \"kubernetes.io/projected/3eb29b61-5760-4a85-8573-a3205e66e833-kube-api-access-p7d2k\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7fhfw\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.375139 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.375102    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:41.667095 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.667012    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"]
Oct 30 19:47:41.670225 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:47:41.670197    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb29b61_5760_4a85_8573_a3205e66e833.slice/crio-2614457c65348f8a3b9afce5574af11aa3241dff590be4ac9a2dbe07d32ab22e WatchSource:0}: Error finding container 2614457c65348f8a3b9afce5574af11aa3241dff590be4ac9a2dbe07d32ab22e: Status 404 returned error can't find the container with id 2614457c65348f8a3b9afce5574af11aa3241dff590be4ac9a2dbe07d32ab22e
Oct 30 19:47:41.672296 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.672265    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:47:41.672355 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.672330    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:47:41.672390 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:41.672356    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:47:42.394458 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:42.394417    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw" event={"ID":"3eb29b61-5760-4a85-8573-a3205e66e833","Type":"ContainerStarted","Data":"cf51afe5888ea55d599df5293174186b3f670f69e2db99b0127ea60cdfd50c6c"}
Oct 30 19:47:42.394976 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:42.394471    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw" event={"ID":"3eb29b61-5760-4a85-8573-a3205e66e833","Type":"ContainerStarted","Data":"2614457c65348f8a3b9afce5574af11aa3241dff590be4ac9a2dbe07d32ab22e"}
Oct 30 19:47:42.429195 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:42.429144    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw" podStartSLOduration=1.429129987 podStartE2EDuration="1.429129987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:42.42719491 +0000 UTC m=+1062.319809245" watchObservedRunningTime="2026-10-30 19:47:42.429129987 +0000 UTC m=+1062.321744308"
Oct 30 19:47:43.377041 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:43.377005    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:43.378690 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:43.378661    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7fhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 19:47:43.378805 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:43.378720    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw" podUID="3eb29b61-5760-4a85-8573-a3205e66e833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 19:47:44.377281 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:44.377240    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7fhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 19:47:44.377812 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:44.377297    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw" podUID="3eb29b61-5760-4a85-8573-a3205e66e833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 19:47:45.377322 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:45.377281    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7fhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 19:47:45.377974 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:45.377936    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw" podUID="3eb29b61-5760-4a85-8573-a3205e66e833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 19:47:46.377325 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:46.377282    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7fhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 19:47:46.377967 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:46.377930    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw" podUID="3eb29b61-5760-4a85-8573-a3205e66e833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 19:47:47.376300 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:47.376262    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7fhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 19:47:47.376598 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:47.376564    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw" podUID="3eb29b61-5760-4a85-8573-a3205e66e833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 19:47:48.380008 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:48.379979    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:48.380498 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:48.380417    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:48.381139 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:48.381119    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 19:47:56.115286 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.115256    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"]
Oct 30 19:47:56.117893 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.117850    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.202395 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.202357    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.202580 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.202416    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.202580 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.202545    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.202693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.202608    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.202693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.202666    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.202757 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.202725    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.202912 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.202896    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.202993 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.202928    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgsjj\" (UniqueName: \"kubernetes.io/projected/11656a86-294a-4cd1-a29c-6f8d02a39c2b-kube-api-access-qgsjj\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.203066 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.203047    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.206273 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.206226    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"]
Oct 30 19:47:56.303752 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.303720    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.303944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.303761    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.303944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.303783    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.303944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.303910    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.304086 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.303964    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.304086 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.304001    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.304086 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.304026    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.304235 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.304115    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.304235 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.304144    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgsjj\" (UniqueName: \"kubernetes.io/projected/11656a86-294a-4cd1-a29c-6f8d02a39c2b-kube-api-access-qgsjj\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.304235 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.304204    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.304388 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.304311    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.304388 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.304346    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.304605 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.304573    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.304715 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.304649    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.306094 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.306074    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.306403 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.306383    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.316883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.316844    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.317156 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.317139    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgsjj\" (UniqueName: \"kubernetes.io/projected/11656a86-294a-4cd1-a29c-6f8d02a39c2b-kube-api-access-qgsjj\") pod \"maas-default-gateway-openshift-default-57f78bcd55-4zvvg\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.430022 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.429949    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:56.720225 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.720200    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"]
Oct 30 19:47:56.722958 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:47:56.722932    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11656a86_294a_4cd1_a29c_6f8d02a39c2b.slice/crio-ab1c3225299b3f9865ca99aed5bc54fb8b4741878fb10872a6cde357aa08f4ff WatchSource:0}: Error finding container ab1c3225299b3f9865ca99aed5bc54fb8b4741878fb10872a6cde357aa08f4ff: Status 404 returned error can't find the container with id ab1c3225299b3f9865ca99aed5bc54fb8b4741878fb10872a6cde357aa08f4ff
Oct 30 19:47:56.725070 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.725041    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:47:56.725129 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.725102    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:47:56.725164 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:56.725129    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:47:57.483242 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:57.483209    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg" event={"ID":"11656a86-294a-4cd1-a29c-6f8d02a39c2b","Type":"ContainerStarted","Data":"9fcfd6ff1c9e699ceba6fe0f726cd6e4da9785667e55519cd9e8291e16dca3d6"}
Oct 30 19:47:57.483653 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:57.483261    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg" event={"ID":"11656a86-294a-4cd1-a29c-6f8d02a39c2b","Type":"ContainerStarted","Data":"ab1c3225299b3f9865ca99aed5bc54fb8b4741878fb10872a6cde357aa08f4ff"}
Oct 30 19:47:57.510056 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:57.510008    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg" podStartSLOduration=1.50999511 podStartE2EDuration="1.50999511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:57.507279453 +0000 UTC m=+1077.399893777" watchObservedRunningTime="2026-10-30 19:47:57.50999511 +0000 UTC m=+1077.402609479"
Oct 30 19:47:58.430936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:58.430898    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:47:58.432547 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:58.432524    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-4zvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 19:47:58.432673 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:58.432569    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg" podUID="11656a86-294a-4cd1-a29c-6f8d02a39c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 19:47:59.430946 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:59.430914    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-4zvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 19:47:59.431339 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:47:59.430968    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg" podUID="11656a86-294a-4cd1-a29c-6f8d02a39c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 19:48:00.430923 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:00.430884    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-4zvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 19:48:00.431091 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:00.430938    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg" podUID="11656a86-294a-4cd1-a29c-6f8d02a39c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 19:48:01.430664 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:01.430632    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-4zvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 19:48:01.430825 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:01.430682    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg" podUID="11656a86-294a-4cd1-a29c-6f8d02a39c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 19:48:02.013194 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:48:02.013154    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:48:02.430423 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:02.430391    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-4zvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 19:48:02.430662 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:02.430457    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg" podUID="11656a86-294a-4cd1-a29c-6f8d02a39c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 19:48:03.434062 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:03.434026    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:48:03.434662 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:03.434415    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:48:03.434992 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:03.434974    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 19:48:09.616927 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.616896    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"]
Oct 30 19:48:09.620975 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.620955    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.626978 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.626955    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 19:48:09.636900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.636875    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"]
Oct 30 19:48:09.640545 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.640523    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 19:48:09.640634 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.640570    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 19:48:09.726585 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.726559    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-z86lw\""
Oct 30 19:48:09.756032 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.756000    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7473cc37-e4fc-44cc-bb8b-4a42432af125-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.756192 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.756055    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7473cc37-e4fc-44cc-bb8b-4a42432af125-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.756192 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.756176    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7473cc37-e4fc-44cc-bb8b-4a42432af125-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.756265 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.756225    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh6b7\" (UniqueName: \"kubernetes.io/projected/7473cc37-e4fc-44cc-bb8b-4a42432af125-kube-api-access-wh6b7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.756318 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.756306    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7473cc37-e4fc-44cc-bb8b-4a42432af125-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.756375 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.756363    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7473cc37-e4fc-44cc-bb8b-4a42432af125-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.857376 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.857337    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7473cc37-e4fc-44cc-bb8b-4a42432af125-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.857546 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.857402    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7473cc37-e4fc-44cc-bb8b-4a42432af125-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.857546 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.857433    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7473cc37-e4fc-44cc-bb8b-4a42432af125-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.857546 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.857463    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh6b7\" (UniqueName: \"kubernetes.io/projected/7473cc37-e4fc-44cc-bb8b-4a42432af125-kube-api-access-wh6b7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.857546 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.857491    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7473cc37-e4fc-44cc-bb8b-4a42432af125-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.857546 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.857543    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7473cc37-e4fc-44cc-bb8b-4a42432af125-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.857829 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.857800    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7473cc37-e4fc-44cc-bb8b-4a42432af125-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.857927 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.857836    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7473cc37-e4fc-44cc-bb8b-4a42432af125-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.858006 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.857985    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7473cc37-e4fc-44cc-bb8b-4a42432af125-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.859818 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.859795    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7473cc37-e4fc-44cc-bb8b-4a42432af125-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.860134 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.860112    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7473cc37-e4fc-44cc-bb8b-4a42432af125-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.874650 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.874601    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh6b7\" (UniqueName: \"kubernetes.io/projected/7473cc37-e4fc-44cc-bb8b-4a42432af125-kube-api-access-wh6b7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs\" (UID: \"7473cc37-e4fc-44cc-bb8b-4a42432af125\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:09.933360 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:09.933325    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:10.088202 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:10.088165    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:48:10.217183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:10.216494    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"]
Oct 30 19:48:10.219337 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:48:10.219286    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7473cc37_e4fc_44cc_bb8b_4a42432af125.slice/crio-95953fbf24f3c6f539332d1116b6c1563160de0441353ba23c27a08f680055f5 WatchSource:0}: Error finding container 95953fbf24f3c6f539332d1116b6c1563160de0441353ba23c27a08f680055f5: Status 404 returned error can't find the container with id 95953fbf24f3c6f539332d1116b6c1563160de0441353ba23c27a08f680055f5
Oct 30 19:48:10.587943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:10.587513    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs" event={"ID":"7473cc37-e4fc-44cc-bb8b-4a42432af125","Type":"ContainerStarted","Data":"95953fbf24f3c6f539332d1116b6c1563160de0441353ba23c27a08f680055f5"}
Oct 30 19:48:11.138318 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.138283    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"]
Oct 30 19:48:11.142135 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.142100    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.159557 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.159521    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"]
Oct 30 19:48:11.285465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.285424    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.285657 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.285475    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.285657 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.285572    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88deb5dd-bf6b-406b-9aa9-103319a550e6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.285759 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.285664    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.285759 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.285696    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.285759 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.285729    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsspr\" (UniqueName: \"kubernetes.io/projected/88deb5dd-bf6b-406b-9aa9-103319a550e6-kube-api-access-zsspr\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.285933 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.285775    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.285933 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.285827    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.286011 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.285997    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.387571 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.387529    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.387752 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.387638    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.387752 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.387669    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.387752 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.387719    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88deb5dd-bf6b-406b-9aa9-103319a550e6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.387956 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.387808    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.387956 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.387837    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.387956 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.387894    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsspr\" (UniqueName: \"kubernetes.io/projected/88deb5dd-bf6b-406b-9aa9-103319a550e6-kube-api-access-zsspr\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.387956 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.387927    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.387956 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.387953    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.388231 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.388062    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.388298 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.388264    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.388388 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.388334    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.388526 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.388475    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88deb5dd-bf6b-406b-9aa9-103319a550e6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.388526 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.388504    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.390639 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.390618    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.390949 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.390928    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.400561 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.400536    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.400838 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.400815    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsspr\" (UniqueName: \"kubernetes.io/projected/88deb5dd-bf6b-406b-9aa9-103319a550e6-kube-api-access-zsspr\") pod \"maas-default-gateway-openshift-default-57f78bcd55-9djkx\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.460731 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.460688    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:11.755748 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.755706    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"]
Oct 30 19:48:11.761502 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:48:11.761458    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88deb5dd_bf6b_406b_9aa9_103319a550e6.slice/crio-080d67cc2f8336682386c3984dd4e4f8c184ffaf7dfead32f8c5f38de2a126b9 WatchSource:0}: Error finding container 080d67cc2f8336682386c3984dd4e4f8c184ffaf7dfead32f8c5f38de2a126b9: Status 404 returned error can't find the container with id 080d67cc2f8336682386c3984dd4e4f8c184ffaf7dfead32f8c5f38de2a126b9
Oct 30 19:48:11.763959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.763929    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:48:11.764052 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.763994    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:48:11.764052 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:11.764019    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:48:12.594749 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:12.594716    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:48:12.598447 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:12.598412    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs" event={"ID":"7473cc37-e4fc-44cc-bb8b-4a42432af125","Type":"ContainerStarted","Data":"7e0207fdf2fbcaa9a012f5132fc0a1ffe5d69668a15ed0b17dccb66ad0522884"}
Oct 30 19:48:12.600104 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:12.600075    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx" event={"ID":"88deb5dd-bf6b-406b-9aa9-103319a550e6","Type":"ContainerStarted","Data":"bb00618104779c69ce0b4df90def3dbf24e91dc460af633bcd25b6686d3dba13"}
Oct 30 19:48:12.600267 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:12.600127    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx" event={"ID":"88deb5dd-bf6b-406b-9aa9-103319a550e6","Type":"ContainerStarted","Data":"080d67cc2f8336682386c3984dd4e4f8c184ffaf7dfead32f8c5f38de2a126b9"}
Oct 30 19:48:12.630101 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:12.630054    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs" podStartSLOduration=2.240845062 podStartE2EDuration="3.630040362s" totalImagesPullingTime="1.3891953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:12.628416963 +0000 UTC m=+1092.521031298" watchObservedRunningTime="2026-10-30 19:48:12.630040362 +0000 UTC m=+1092.522654685"
Oct 30 19:48:12.676923 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:12.653952    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx" podStartSLOduration=1.6539342769999998 podStartE2EDuration="1.653934277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:12.650962884 +0000 UTC m=+1092.543577207" watchObservedRunningTime="2026-10-30 19:48:12.653934277 +0000 UTC m=+1092.546548603"
Oct 30 19:48:12.933999 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:12.933956    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:13.465948 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:13.465851    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:13.468100 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:13.468067    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-9djkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 19:48:13.468731 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:13.468449    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx" podUID="88deb5dd-bf6b-406b-9aa9-103319a550e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 19:48:14.469293 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:14.469254    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-9djkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 19:48:14.469918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:14.469885    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx" podUID="88deb5dd-bf6b-406b-9aa9-103319a550e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 19:48:14.814569 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:14.814495    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:48:15.469886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:15.466948    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-9djkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 19:48:15.469886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:15.467011    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx" podUID="88deb5dd-bf6b-406b-9aa9-103319a550e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 19:48:16.412436 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.412408    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"]
Oct 30 19:48:16.416091 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.416071    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.431998 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.431974    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 19:48:16.434954 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.434931    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"]
Oct 30 19:48:16.476468 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.476430    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-9djkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 19:48:16.477607 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.477030    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx" podUID="88deb5dd-bf6b-406b-9aa9-103319a550e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 19:48:16.479509 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.479435    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbq2r\" (UniqueName: \"kubernetes.io/projected/44597884-0bca-4006-a0cd-6bf9344678bc-kube-api-access-sbq2r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.479509 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.479493    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44597884-0bca-4006-a0cd-6bf9344678bc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.479683 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.479541    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44597884-0bca-4006-a0cd-6bf9344678bc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.479683 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.479567    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44597884-0bca-4006-a0cd-6bf9344678bc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.479683 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.479594    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44597884-0bca-4006-a0cd-6bf9344678bc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.479683 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.479666    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44597884-0bca-4006-a0cd-6bf9344678bc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.583887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.583419    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbq2r\" (UniqueName: \"kubernetes.io/projected/44597884-0bca-4006-a0cd-6bf9344678bc-kube-api-access-sbq2r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.583887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.583480    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44597884-0bca-4006-a0cd-6bf9344678bc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.583887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.583531    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44597884-0bca-4006-a0cd-6bf9344678bc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.583887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.583559    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44597884-0bca-4006-a0cd-6bf9344678bc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.583887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.583585    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44597884-0bca-4006-a0cd-6bf9344678bc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.583887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.583654    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44597884-0bca-4006-a0cd-6bf9344678bc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.584355 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.584214    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44597884-0bca-4006-a0cd-6bf9344678bc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.588909 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.584778    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44597884-0bca-4006-a0cd-6bf9344678bc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.588909 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.586476    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44597884-0bca-4006-a0cd-6bf9344678bc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.592887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.592664    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44597884-0bca-4006-a0cd-6bf9344678bc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.593617 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.593579    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44597884-0bca-4006-a0cd-6bf9344678bc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.597563 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.597535    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbq2r\" (UniqueName: \"kubernetes.io/projected/44597884-0bca-4006-a0cd-6bf9344678bc-kube-api-access-sbq2r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv\" (UID: \"44597884-0bca-4006-a0cd-6bf9344678bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:16.730610 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:16.730525    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:17.073942 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:17.064544    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"]
Oct 30 19:48:17.466770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:17.466736    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-9djkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 19:48:17.467037 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:17.467007    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx" podUID="88deb5dd-bf6b-406b-9aa9-103319a550e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 19:48:17.677360 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:17.676321    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv" event={"ID":"44597884-0bca-4006-a0cd-6bf9344678bc","Type":"ContainerStarted","Data":"6809a4a0b4ccc17c994f08eea7ed12af988ec31fe52a1b382bbadf1a4a83d155"}
Oct 30 19:48:17.730979 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:17.730944    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:17.831777 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:17.831713    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv" podStartSLOduration=1.495412963 podStartE2EDuration="1.831692401s" totalImagesPullingTime="336.279438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:17.713034719 +0000 UTC m=+1097.605649045" watchObservedRunningTime="2026-10-30 19:48:17.831692401 +0000 UTC m=+1097.724306728"
Oct 30 19:48:17.851000 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:17.850932    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2wrpc"]
Oct 30 19:48:17.898885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:17.898099    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2wrpc"]
Oct 30 19:48:17.898885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:17.898268    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:17.922286 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:17.902851    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 30 19:48:18.041961 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.038407    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b23237-c311-4128-93c5-d6d8a3a53612-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.041961 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.038467    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b23237-c311-4128-93c5-d6d8a3a53612-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.041961 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.038500    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b23237-c311-4128-93c5-d6d8a3a53612-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.041961 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.038580    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2msn\" (UniqueName: \"kubernetes.io/projected/c2b23237-c311-4128-93c5-d6d8a3a53612-kube-api-access-n2msn\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.041961 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.038616    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b23237-c311-4128-93c5-d6d8a3a53612-home\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.041961 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.038681    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b23237-c311-4128-93c5-d6d8a3a53612-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.141143 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.140240    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b23237-c311-4128-93c5-d6d8a3a53612-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.141143 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.140311    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b23237-c311-4128-93c5-d6d8a3a53612-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.141143 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.140343    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b23237-c311-4128-93c5-d6d8a3a53612-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.141143 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.140441    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2msn\" (UniqueName: \"kubernetes.io/projected/c2b23237-c311-4128-93c5-d6d8a3a53612-kube-api-access-n2msn\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.141143 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.140480    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b23237-c311-4128-93c5-d6d8a3a53612-home\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.141143 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.140557    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b23237-c311-4128-93c5-d6d8a3a53612-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.141143 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.141045    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b23237-c311-4128-93c5-d6d8a3a53612-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.141680 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.141660    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2b23237-c311-4128-93c5-d6d8a3a53612-home\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.142192 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.142144    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2b23237-c311-4128-93c5-d6d8a3a53612-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.146357 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.144906    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2b23237-c311-4128-93c5-d6d8a3a53612-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.146798 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.146731    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2b23237-c311-4128-93c5-d6d8a3a53612-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.175599 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.175527    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2msn\" (UniqueName: \"kubernetes.io/projected/c2b23237-c311-4128-93c5-d6d8a3a53612-kube-api-access-n2msn\") pod \"echo-simulator-kserve-64b6f8f96c-2wrpc\" (UID: \"c2b23237-c311-4128-93c5-d6d8a3a53612\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.422772 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.422729    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:18.686924 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.686829    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv" event={"ID":"44597884-0bca-4006-a0cd-6bf9344678bc","Type":"ContainerStarted","Data":"d7e45b566189c9b7de7fd3cf8276b28b704471428ea0b7a3b665dcd9bf6916b1"}
Oct 30 19:48:18.708759 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.708733    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2wrpc"]
Oct 30 19:48:18.711559 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:48:18.711527    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b23237_c311_4128_93c5_d6d8a3a53612.slice/crio-bee46c36e3432dfffc0e1b90940e90c336156a0d2d198ee15b2c835088cbf4ef WatchSource:0}: Error finding container bee46c36e3432dfffc0e1b90940e90c336156a0d2d198ee15b2c835088cbf4ef: Status 404 returned error can't find the container with id bee46c36e3432dfffc0e1b90940e90c336156a0d2d198ee15b2c835088cbf4ef
Oct 30 19:48:18.824194 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:18.824157    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:48:19.462225 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:19.462192    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-9djkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:48:19.462427 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:19.462245    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx" podUID="88deb5dd-bf6b-406b-9aa9-103319a550e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:19.698004 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:19.697940    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc" event={"ID":"c2b23237-c311-4128-93c5-d6d8a3a53612","Type":"ContainerStarted","Data":"bee46c36e3432dfffc0e1b90940e90c336156a0d2d198ee15b2c835088cbf4ef"}
Oct 30 19:48:19.935204 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:19.934427    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:19.957522 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:19.957230    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:20.494021 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:20.493981    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-9djkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:48:20.494608 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:20.494573    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx" podUID="88deb5dd-bf6b-406b-9aa9-103319a550e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:20.529215 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:20.529176    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:20.531458 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:20.531432    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:20.559161 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:20.558184    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 19:48:20.941887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:20.941274    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs"
Oct 30 19:48:21.300945 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.278541    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"]
Oct 30 19:48:21.340127 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.340094    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"]
Oct 30 19:48:21.340443 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.340420    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.401900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.376051    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/531e5023-637b-4a75-9846-8e53b962fc14-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.401900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.376117    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/531e5023-637b-4a75-9846-8e53b962fc14-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.401900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.376157    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/531e5023-637b-4a75-9846-8e53b962fc14-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.401900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.376228    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/531e5023-637b-4a75-9846-8e53b962fc14-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.401900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.376276    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkftq\" (UniqueName: \"kubernetes.io/projected/531e5023-637b-4a75-9846-8e53b962fc14-kube-api-access-xkftq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.401900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.376311    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/531e5023-637b-4a75-9846-8e53b962fc14-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.401900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.376919    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 19:48:21.494918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.486358    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/531e5023-637b-4a75-9846-8e53b962fc14-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.494918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.489127    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/531e5023-637b-4a75-9846-8e53b962fc14-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.494918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.489645    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/531e5023-637b-4a75-9846-8e53b962fc14-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.494918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.489812    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/531e5023-637b-4a75-9846-8e53b962fc14-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.494918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.490186    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/531e5023-637b-4a75-9846-8e53b962fc14-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.494918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.490265    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkftq\" (UniqueName: \"kubernetes.io/projected/531e5023-637b-4a75-9846-8e53b962fc14-kube-api-access-xkftq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.494918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.490565    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/531e5023-637b-4a75-9846-8e53b962fc14-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.494918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.490633    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/531e5023-637b-4a75-9846-8e53b962fc14-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.494918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.491567    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/531e5023-637b-4a75-9846-8e53b962fc14-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.508250 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.508196    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/531e5023-637b-4a75-9846-8e53b962fc14-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.523065 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.523002    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkftq\" (UniqueName: \"kubernetes.io/projected/531e5023-637b-4a75-9846-8e53b962fc14-kube-api-access-xkftq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.525708 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.525651    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/531e5023-637b-4a75-9846-8e53b962fc14-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw\" (UID: \"531e5023-637b-4a75-9846-8e53b962fc14\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:21.799530 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:21.799496    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:22.112953 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:22.112919    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"]
Oct 30 19:48:22.117316 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:48:22.117192    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod531e5023_637b_4a75_9846_8e53b962fc14.slice/crio-d71cd740abc0f212c78dad023045217615f906781232d80bcaa5d4fe368a5d78 WatchSource:0}: Error finding container d71cd740abc0f212c78dad023045217615f906781232d80bcaa5d4fe368a5d78: Status 404 returned error can't find the container with id d71cd740abc0f212c78dad023045217615f906781232d80bcaa5d4fe368a5d78
Oct 30 19:48:22.824543 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:22.824368    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" event={"ID":"531e5023-637b-4a75-9846-8e53b962fc14","Type":"ContainerStarted","Data":"82cc9fc5125e828b6cba1d43bf36f4e06eb349c92528c1efb93099f2db3515e5"}
Oct 30 19:48:22.824543 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:22.824441    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" event={"ID":"531e5023-637b-4a75-9846-8e53b962fc14","Type":"ContainerStarted","Data":"d71cd740abc0f212c78dad023045217615f906781232d80bcaa5d4fe368a5d78"}
Oct 30 19:48:22.826936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:22.826901    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc" event={"ID":"c2b23237-c311-4128-93c5-d6d8a3a53612","Type":"ContainerStarted","Data":"23bb88a540f1de7896920b86c31792bec06d95415a1afd64d523b28558f211c1"}
Oct 30 19:48:22.845900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:22.845825    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podStartSLOduration=1.543352258 podStartE2EDuration="1.845811953s" totalImagesPullingTime="302.459695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:22.843667844 +0000 UTC m=+1102.736282184" watchObservedRunningTime="2026-10-30 19:48:22.845811953 +0000 UTC m=+1102.738426276"
Oct 30 19:48:22.870362 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:22.870303    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc" podStartSLOduration=2.223020489 podStartE2EDuration="5.870285321s" totalImagesPullingTime="3.647264832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:22.868527707 +0000 UTC m=+1102.761142058" watchObservedRunningTime="2026-10-30 19:48:22.870285321 +0000 UTC m=+1102.762899646"
Oct 30 19:48:23.423380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:23.423344    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:23.803507 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:23.801950    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:26.138325 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:26.138285    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:48:26.730927 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:26.730894    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:26.743649 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:26.743622    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:26.884558 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:26.884526    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv"
Oct 30 19:48:28.423386 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:28.423344    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:28.436285 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:28.436255    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:28.894593 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:28.894567    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc"
Oct 30 19:48:31.800265 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:31.800233    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:31.812746 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:31.812723    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:31.910131 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:31.910099    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:48:32.012831 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:48:32.012795    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:48:45.248653 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:45.248577    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-p592q"]
Oct 30 19:48:45.259677 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:45.259649    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p592q"
Oct 30 19:48:45.265436 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:45.265405    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-p592q"]
Oct 30 19:48:45.273002 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:45.272975    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l4qwx\""
Oct 30 19:48:45.273341 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:45.273318    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 19:48:45.387918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:45.374779    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbrxt\" (UniqueName: \"kubernetes.io/projected/d88a34ee-0b5d-4d35-b1bf-1679f82aab9f-kube-api-access-zbrxt\") pod \"payload-pre-processing-7b98bb678c-p592q\" (UID: \"d88a34ee-0b5d-4d35-b1bf-1679f82aab9f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p592q"
Oct 30 19:48:45.388616 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:45.388582    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88a34ee-0b5d-4d35-b1bf-1679f82aab9f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-p592q\" (UID: \"d88a34ee-0b5d-4d35-b1bf-1679f82aab9f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p592q"
Oct 30 19:48:45.489658 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:45.489617    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbrxt\" (UniqueName: \"kubernetes.io/projected/d88a34ee-0b5d-4d35-b1bf-1679f82aab9f-kube-api-access-zbrxt\") pod \"payload-pre-processing-7b98bb678c-p592q\" (UID: \"d88a34ee-0b5d-4d35-b1bf-1679f82aab9f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p592q"
Oct 30 19:48:45.489855 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:45.489702    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88a34ee-0b5d-4d35-b1bf-1679f82aab9f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-p592q\" (UID: \"d88a34ee-0b5d-4d35-b1bf-1679f82aab9f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p592q"
Oct 30 19:48:45.490489 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:45.490467    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d88a34ee-0b5d-4d35-b1bf-1679f82aab9f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-p592q\" (UID: \"d88a34ee-0b5d-4d35-b1bf-1679f82aab9f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p592q"
Oct 30 19:48:45.514793 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:45.514725    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbrxt\" (UniqueName: \"kubernetes.io/projected/d88a34ee-0b5d-4d35-b1bf-1679f82aab9f-kube-api-access-zbrxt\") pod \"payload-pre-processing-7b98bb678c-p592q\" (UID: \"d88a34ee-0b5d-4d35-b1bf-1679f82aab9f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p592q"
Oct 30 19:48:45.577759 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:45.577725    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p592q"
Oct 30 19:48:45.872353 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:45.872327    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-p592q"]
Oct 30 19:48:45.874109 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:48:45.874080    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88a34ee_0b5d_4d35_b1bf_1679f82aab9f.slice/crio-b5b628d53d30650b745c3b481bc9fdcd1999872e9fb21a65597187e37883f198 WatchSource:0}: Error finding container b5b628d53d30650b745c3b481bc9fdcd1999872e9fb21a65597187e37883f198: Status 404 returned error can't find the container with id b5b628d53d30650b745c3b481bc9fdcd1999872e9fb21a65597187e37883f198
Oct 30 19:48:45.970827 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:45.970795    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p592q" event={"ID":"d88a34ee-0b5d-4d35-b1bf-1679f82aab9f","Type":"ContainerStarted","Data":"b5b628d53d30650b745c3b481bc9fdcd1999872e9fb21a65597187e37883f198"}
Oct 30 19:48:48.989123 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:48.989079    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p592q" event={"ID":"d88a34ee-0b5d-4d35-b1bf-1679f82aab9f","Type":"ContainerStarted","Data":"7eb1c8bacde9ec7484462d6c4226ffc3595d56c15e1c2c52d76bcb54bec95870"}
Oct 30 19:48:48.989585 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:48.989140    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p592q"
Oct 30 19:48:49.016315 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:49.016259    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p592q" podStartSLOduration=1.868874294 podStartE2EDuration="4.01624628s" totalImagesPullingTime="2.147371986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:49.016032809 +0000 UTC m=+1128.908647133" watchObservedRunningTime="2026-10-30 19:48:49.01624628 +0000 UTC m=+1128.908860604"
Oct 30 19:48:54.933041 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:54.933004    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7b9fb75546-dcfw8"]
Oct 30 19:48:54.936966 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:54.936947    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7b9fb75546-dcfw8"
Oct 30 19:48:54.945309 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:54.945288    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 19:48:54.945562 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:54.945545    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 19:48:54.945562 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:54.945554    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-f4tgk\""
Oct 30 19:48:54.946152 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:54.946120    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7b9fb75546-dcfw8"]
Oct 30 19:48:54.978639 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:54.978604    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48kkn\" (UniqueName: \"kubernetes.io/projected/059a7cf4-196d-4ed8-b1d0-654d6e521c2a-kube-api-access-48kkn\") pod \"ai-gateway-controller-7b9fb75546-dcfw8\" (UID: \"059a7cf4-196d-4ed8-b1d0-654d6e521c2a\") " pod="opendatahub/ai-gateway-controller-7b9fb75546-dcfw8"
Oct 30 19:48:54.978813 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:54.978674    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a7cf4-196d-4ed8-b1d0-654d6e521c2a-metrics-certs\") pod \"ai-gateway-controller-7b9fb75546-dcfw8\" (UID: \"059a7cf4-196d-4ed8-b1d0-654d6e521c2a\") " pod="opendatahub/ai-gateway-controller-7b9fb75546-dcfw8"
Oct 30 19:48:55.079270 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:55.079236    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48kkn\" (UniqueName: \"kubernetes.io/projected/059a7cf4-196d-4ed8-b1d0-654d6e521c2a-kube-api-access-48kkn\") pod \"ai-gateway-controller-7b9fb75546-dcfw8\" (UID: \"059a7cf4-196d-4ed8-b1d0-654d6e521c2a\") " pod="opendatahub/ai-gateway-controller-7b9fb75546-dcfw8"
Oct 30 19:48:55.079463 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:55.079381    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a7cf4-196d-4ed8-b1d0-654d6e521c2a-metrics-certs\") pod \"ai-gateway-controller-7b9fb75546-dcfw8\" (UID: \"059a7cf4-196d-4ed8-b1d0-654d6e521c2a\") " pod="opendatahub/ai-gateway-controller-7b9fb75546-dcfw8"
Oct 30 19:48:55.082214 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:55.082188    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/059a7cf4-196d-4ed8-b1d0-654d6e521c2a-metrics-certs\") pod \"ai-gateway-controller-7b9fb75546-dcfw8\" (UID: \"059a7cf4-196d-4ed8-b1d0-654d6e521c2a\") " pod="opendatahub/ai-gateway-controller-7b9fb75546-dcfw8"
Oct 30 19:48:55.091588 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:55.091563    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48kkn\" (UniqueName: \"kubernetes.io/projected/059a7cf4-196d-4ed8-b1d0-654d6e521c2a-kube-api-access-48kkn\") pod \"ai-gateway-controller-7b9fb75546-dcfw8\" (UID: \"059a7cf4-196d-4ed8-b1d0-654d6e521c2a\") " pod="opendatahub/ai-gateway-controller-7b9fb75546-dcfw8"
Oct 30 19:48:55.250577 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:55.250487    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7b9fb75546-dcfw8"
Oct 30 19:48:55.531430 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:55.531406    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7b9fb75546-dcfw8"]
Oct 30 19:48:55.533823 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:48:55.533798    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059a7cf4_196d_4ed8_b1d0_654d6e521c2a.slice/crio-1d64404da9a48570c973f61bbced1fa03992123519058498778cb25efe7bee5d WatchSource:0}: Error finding container 1d64404da9a48570c973f61bbced1fa03992123519058498778cb25efe7bee5d: Status 404 returned error can't find the container with id 1d64404da9a48570c973f61bbced1fa03992123519058498778cb25efe7bee5d
Oct 30 19:48:56.025654 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:56.025620    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7b9fb75546-dcfw8" event={"ID":"059a7cf4-196d-4ed8-b1d0-654d6e521c2a","Type":"ContainerStarted","Data":"1d64404da9a48570c973f61bbced1fa03992123519058498778cb25efe7bee5d"}
Oct 30 19:48:58.047755 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:58.047719    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7b9fb75546-dcfw8" event={"ID":"059a7cf4-196d-4ed8-b1d0-654d6e521c2a","Type":"ContainerStarted","Data":"0c82d5c5ff246477897d035e9d671ea297ada1b69a522386f45fdf1dc15d0b2c"}
Oct 30 19:48:58.048143 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:58.047801    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7b9fb75546-dcfw8"
Oct 30 19:48:58.067379 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:58.067334    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7b9fb75546-dcfw8" podStartSLOduration=1.644588305 podStartE2EDuration="4.067321305s" totalImagesPullingTime="2.422733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:58.067299862 +0000 UTC m=+1137.959914187" watchObservedRunningTime="2026-10-30 19:48:58.067321305 +0000 UTC m=+1137.959935625"
Oct 30 19:48:59.997481 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:48:59.997446    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p592q"
Oct 30 19:49:02.013826 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:49:02.013787    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:49:09.055402 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:09.055375    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7b9fb75546-dcfw8"
Oct 30 19:49:32.013410 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:49:32.013370    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:49:39.536592 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.536557    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"]
Oct 30 19:49:39.543790 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.543766    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.553492 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.553468    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d5e3e5-openshift-default-dockercfg-pw4z9\""
Oct 30 19:49:39.562681 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.562655    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"]
Oct 30 19:49:39.621136 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.621093    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.621387 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.621153    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091b9ee1-723f-44d7-bac5-629178dbd234-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.621387 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.621187    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.621387 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.621265    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091b9ee1-723f-44d7-bac5-629178dbd234-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.621387 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.621354    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.621620 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.621418    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd8dv\" (UniqueName: \"kubernetes.io/projected/091b9ee1-723f-44d7-bac5-629178dbd234-kube-api-access-fd8dv\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.621620 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.621507    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.621620 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.621601    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.621763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.621624    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091b9ee1-723f-44d7-bac5-629178dbd234-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.723117 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.723082    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.723312 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.723149    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.723312 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.723168    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091b9ee1-723f-44d7-bac5-629178dbd234-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.723312 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.723201    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.723312 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.723229    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091b9ee1-723f-44d7-bac5-629178dbd234-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.723312 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.723248    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.723312 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.723272    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091b9ee1-723f-44d7-bac5-629178dbd234-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.723616 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.723316    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.723616 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.723378    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd8dv\" (UniqueName: \"kubernetes.io/projected/091b9ee1-723f-44d7-bac5-629178dbd234-kube-api-access-fd8dv\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.723616 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.723541    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.723776 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.723627    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.723836 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.723808    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.723942 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.723921    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.723995 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.723964    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091b9ee1-723f-44d7-bac5-629178dbd234-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.725683 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.725659    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.725898 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.725855    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091b9ee1-723f-44d7-bac5-629178dbd234-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.750458 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.750394    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd8dv\" (UniqueName: \"kubernetes.io/projected/091b9ee1-723f-44d7-bac5-629178dbd234-kube-api-access-fd8dv\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.755230 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.755197    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091b9ee1-723f-44d7-bac5-629178dbd234-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.857495 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.857464    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:39.923418 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.923221    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"]
Oct 30 19:49:39.927808 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.927784    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:39.946326 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.946303    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ced832-openshift-default-dockercfg-4thzg\""
Oct 30 19:49:39.952624 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:39.952598    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"]
Oct 30 19:49:40.025394 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.025357    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd984f9d-dcce-452e-a843-e07894b05c69-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.025394 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.025401    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.025641 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.025417    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd984f9d-dcce-452e-a843-e07894b05c69-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.025641 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.025572    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd984f9d-dcce-452e-a843-e07894b05c69-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.025729 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.025678    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.025804 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.025781    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df4gh\" (UniqueName: \"kubernetes.io/projected/dd984f9d-dcce-452e-a843-e07894b05c69-kube-api-access-df4gh\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.025997 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.025978    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.026078 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.026007    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.026078 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.026052    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.128092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.127168    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd984f9d-dcce-452e-a843-e07894b05c69-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.128092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.127216    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.128092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.127301    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df4gh\" (UniqueName: \"kubernetes.io/projected/dd984f9d-dcce-452e-a843-e07894b05c69-kube-api-access-df4gh\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.128092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.127336    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.128092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.127359    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.128092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.127410    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.128092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.127488    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd984f9d-dcce-452e-a843-e07894b05c69-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.128092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.127533    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.128092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.127559    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd984f9d-dcce-452e-a843-e07894b05c69-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.128985 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.128957    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.129142 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.129011    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.129499 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.129475    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.129760 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.129737    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.130001 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.129969    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd984f9d-dcce-452e-a843-e07894b05c69-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.132324 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.132304    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.132436 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.132416    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd984f9d-dcce-452e-a843-e07894b05c69-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.149163 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.149139    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd984f9d-dcce-452e-a843-e07894b05c69-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.150065 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.150043    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df4gh\" (UniqueName: \"kubernetes.io/projected/dd984f9d-dcce-452e-a843-e07894b05c69-kube-api-access-df4gh\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.163338 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.163305    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"]
Oct 30 19:49:40.166323 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:49:40.166297    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091b9ee1_723f_44d7_bac5_629178dbd234.slice/crio-080255860f3e39bf5aef44b90ca1987ece5cc6558ed1e6737007b1294cb28715 WatchSource:0}: Error finding container 080255860f3e39bf5aef44b90ca1987ece5cc6558ed1e6737007b1294cb28715: Status 404 returned error can't find the container with id 080255860f3e39bf5aef44b90ca1987ece5cc6558ed1e6737007b1294cb28715
Oct 30 19:49:40.170294 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.170259    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:49:40.170401 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.170342    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:49:40.170401 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.170388    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:49:40.257443 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.255907    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:40.257443 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.257297    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"]
Oct 30 19:49:40.260362 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.260334    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.268159 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.268125    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws" event={"ID":"091b9ee1-723f-44d7-bac5-629178dbd234","Type":"ContainerStarted","Data":"16c71fad5a5ad82b8bf869736e059b942aff2723a3249185755840d3110a9b79"}
Oct 30 19:49:40.268298 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.268188    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws" event={"ID":"091b9ee1-723f-44d7-bac5-629178dbd234","Type":"ContainerStarted","Data":"080255860f3e39bf5aef44b90ca1987ece5cc6558ed1e6737007b1294cb28715"}
Oct 30 19:49:40.278028 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.278004    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-278ffa-openshift-default-dockercfg-9bwm7\""
Oct 30 19:49:40.299954 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.298622    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"]
Oct 30 19:49:40.330726 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.330676    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.331002 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.330969    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.331102 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.331028    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.331102 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.331053    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.331217 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.331175    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.331272 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.331214    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.331272 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.331243    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.331377 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.331271    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9pr\" (UniqueName: \"kubernetes.io/projected/82f08656-55bf-4de5-9beb-3776bc8dd3f2-kube-api-access-4t9pr\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.331431 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.331407    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.417787 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.417671    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws" podStartSLOduration=1.4176510580000001 podStartE2EDuration="1.417651058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:40.416127756 +0000 UTC m=+1180.308742081" watchObservedRunningTime="2026-10-30 19:49:40.417651058 +0000 UTC m=+1180.310265383"
Oct 30 19:49:40.435148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.432919    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.435148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.433669    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.435148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.433742    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.435148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.433765    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.435148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.434028    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.435148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.434071    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9pr\" (UniqueName: \"kubernetes.io/projected/82f08656-55bf-4de5-9beb-3776bc8dd3f2-kube-api-access-4t9pr\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.435148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.434267    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.435148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.434327    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.435148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.434814    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.435148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.434881    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.435148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.434909    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.435148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.434937    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.435148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.434953    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.435992 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.435785    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.441905 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.438139    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.441905 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.439691    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.447171 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.447143    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9pr\" (UniqueName: \"kubernetes.io/projected/82f08656-55bf-4de5-9beb-3776bc8dd3f2-kube-api-access-4t9pr\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.454075 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.454026    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.582628 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.582592    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:40.641586 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.641552    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"]
Oct 30 19:49:40.652552 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.652508    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:49:40.652644 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.652597    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:49:40.652688 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.652642    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:49:40.755979 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.755933    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"]
Oct 30 19:49:40.760372 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.760350    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.772558 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.772535    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8a8207-openshift-default-dockercfg-shmd6\""
Oct 30 19:49:40.840578 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.840539    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.840578 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.840580    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2cps\" (UniqueName: \"kubernetes.io/projected/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-kube-api-access-d2cps\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.840854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.840639    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.840854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.840668    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.840854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.840693    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.840854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.840741    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.840854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.840779    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.840854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.840801    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.840854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.840823    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.847949 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.847619    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"]
Oct 30 19:49:40.858212 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.858182    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:40.933803 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.933760    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"]
Oct 30 19:49:40.936727 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:49:40.936283    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f08656_55bf_4de5_9beb_3776bc8dd3f2.slice/crio-2995c2de98c2851b089489e5db0c5549a99bf6d9cf7b3fa7f000b7045cfd9cd8 WatchSource:0}: Error finding container 2995c2de98c2851b089489e5db0c5549a99bf6d9cf7b3fa7f000b7045cfd9cd8: Status 404 returned error can't find the container with id 2995c2de98c2851b089489e5db0c5549a99bf6d9cf7b3fa7f000b7045cfd9cd8
Oct 30 19:49:40.938902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.938835    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:49:40.939005 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.938924    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:49:40.939005 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.938968    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:49:40.947271 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.947243    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.947397 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.947298    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.947397 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.947329    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.947397 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.947380    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.947573 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.947428    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.947573 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.947457    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.947573 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.947490    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.947573 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.947537    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.947573 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.947559    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2cps\" (UniqueName: \"kubernetes.io/projected/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-kube-api-access-d2cps\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.948641 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.948585    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.948747 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.948655    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.949040 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.948999    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.949186 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.949071    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.949516 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.949464    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.952047 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.952020    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.952218 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.952199    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.965117 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.965092    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2cps\" (UniqueName: \"kubernetes.io/projected/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-kube-api-access-d2cps\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:40.966551 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:40.966354    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:41.092454 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:41.092410    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:41.275804 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:41.275722    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44" event={"ID":"dd984f9d-dcce-452e-a843-e07894b05c69","Type":"ContainerStarted","Data":"91ce26b56cb6dc68cd1b497564dafeac233a8113118979450423f525660b5e3c"}
Oct 30 19:49:41.275804 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:41.275780    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44" event={"ID":"dd984f9d-dcce-452e-a843-e07894b05c69","Type":"ContainerStarted","Data":"9815397033e9a6077131dc6568140947a84716f0b5113ad0bb0553a53fd546f0"}
Oct 30 19:49:41.277439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:41.277413    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm" event={"ID":"82f08656-55bf-4de5-9beb-3776bc8dd3f2","Type":"ContainerStarted","Data":"2208641275ce75519737c1cd9455c82a180c032775afbfcf7b62c1002c93a7e6"}
Oct 30 19:49:41.277552 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:41.277458    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm" event={"ID":"82f08656-55bf-4de5-9beb-3776bc8dd3f2","Type":"ContainerStarted","Data":"2995c2de98c2851b089489e5db0c5549a99bf6d9cf7b3fa7f000b7045cfd9cd8"}
Oct 30 19:49:41.326150 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:41.326098    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44" podStartSLOduration=2.326082894 podStartE2EDuration="2.326082894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:41.324150186 +0000 UTC m=+1181.216764512" watchObservedRunningTime="2026-10-30 19:49:41.326082894 +0000 UTC m=+1181.218697217"
Oct 30 19:49:41.392590 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:41.392530    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm" podStartSLOduration=1.392514706 podStartE2EDuration="1.392514706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:41.36288923 +0000 UTC m=+1181.255503544" watchObservedRunningTime="2026-10-30 19:49:41.392514706 +0000 UTC m=+1181.285129030"
Oct 30 19:49:41.394262 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:41.394240    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"]
Oct 30 19:49:41.397474 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:49:41.397448    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda5b0d8_6d21_4ff4_a860_d0f63f912ee0.slice/crio-4f7d73fe6ab8962578a58720f75ac10ca6f31b503ce071c6b74b33d3367d42db WatchSource:0}: Error finding container 4f7d73fe6ab8962578a58720f75ac10ca6f31b503ce071c6b74b33d3367d42db: Status 404 returned error can't find the container with id 4f7d73fe6ab8962578a58720f75ac10ca6f31b503ce071c6b74b33d3367d42db
Oct 30 19:49:41.399836 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:41.399804    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:49:41.399974 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:41.399887    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:49:41.399974 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:41.399918    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:49:41.582971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:41.582898    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:41.863089 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:41.863066    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:42.257023 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:42.256453    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:42.263099 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:42.263072    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:42.284602 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:42.284568    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd" event={"ID":"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0","Type":"ContainerStarted","Data":"c9ff1fae34a53e9dd981298a9664a6cf4fa8b421897d3b40e80b50625cd56853"}
Oct 30 19:49:42.284771 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:42.284621    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd" event={"ID":"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0","Type":"ContainerStarted","Data":"4f7d73fe6ab8962578a58720f75ac10ca6f31b503ce071c6b74b33d3367d42db"}
Oct 30 19:49:42.285322 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:42.285281    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:42.285490 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:42.285472    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:42.285603 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:42.285558    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:49:42.285603 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:42.285593    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:49:42.387141 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:42.387064    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd" podStartSLOduration=2.387042419 podStartE2EDuration="2.387042419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:42.384961088 +0000 UTC m=+1182.277575415" watchObservedRunningTime="2026-10-30 19:49:42.387042419 +0000 UTC m=+1182.279656755"
Oct 30 19:49:42.588485 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:42.588407    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:43.093517 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:43.093484    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:43.098480 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:43.098450    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:43.290642 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:43.290610    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:43.290846 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:43.290665    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:43.291238 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:43.291213    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:49:43.291447 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:43.291421    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:49:46.221550 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:46.221516    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"]
Oct 30 19:49:46.224727 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:46.224706    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"
Oct 30 19:49:46.232059 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:46.232037    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-a1d885-dockercfg-x88z6\""
Oct 30 19:49:46.240405 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:46.240383    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a1d885\""
Oct 30 19:49:46.242185 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:46.242162    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"]
Oct 30 19:49:46.312882 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:46.304796    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0d4ac0-27c1-4735-b333-63b4e75afe70-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm\" (UID: \"fd0d4ac0-27c1-4735-b333-63b4e75afe70\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"
Oct 30 19:49:46.312882 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:46.304923    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dl6n\" (UniqueName: \"kubernetes.io/projected/fd0d4ac0-27c1-4735-b333-63b4e75afe70-kube-api-access-6dl6n\") pod \"payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm\" (UID: \"fd0d4ac0-27c1-4735-b333-63b4e75afe70\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"
Oct 30 19:49:46.406180 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:46.406131    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0d4ac0-27c1-4735-b333-63b4e75afe70-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm\" (UID: \"fd0d4ac0-27c1-4735-b333-63b4e75afe70\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"
Oct 30 19:49:46.406357 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:46.406213    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dl6n\" (UniqueName: \"kubernetes.io/projected/fd0d4ac0-27c1-4735-b333-63b4e75afe70-kube-api-access-6dl6n\") pod \"payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm\" (UID: \"fd0d4ac0-27c1-4735-b333-63b4e75afe70\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"
Oct 30 19:49:46.407241 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:46.407213    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0d4ac0-27c1-4735-b333-63b4e75afe70-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm\" (UID: \"fd0d4ac0-27c1-4735-b333-63b4e75afe70\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"
Oct 30 19:49:46.422161 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:46.422120    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dl6n\" (UniqueName: \"kubernetes.io/projected/fd0d4ac0-27c1-4735-b333-63b4e75afe70-kube-api-access-6dl6n\") pod \"payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm\" (UID: \"fd0d4ac0-27c1-4735-b333-63b4e75afe70\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"
Oct 30 19:49:46.538076 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:46.537987    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"
Oct 30 19:49:46.829645 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:46.829621    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"]
Oct 30 19:49:46.832615 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:49:46.832587    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0d4ac0_27c1_4735_b333_63b4e75afe70.slice/crio-f08ada36b720a6941ccb3e8594a40f3cc5dfe8e2df5876cbf3657ce2a5ca8b1f WatchSource:0}: Error finding container f08ada36b720a6941ccb3e8594a40f3cc5dfe8e2df5876cbf3657ce2a5ca8b1f: Status 404 returned error can't find the container with id f08ada36b720a6941ccb3e8594a40f3cc5dfe8e2df5876cbf3657ce2a5ca8b1f
Oct 30 19:49:47.344106 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:47.344065    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm" event={"ID":"fd0d4ac0-27c1-4735-b333-63b4e75afe70","Type":"ContainerStarted","Data":"635dad2452b6b4b06f14eb51e27df8beae0404ddccb230204de15dff6d17c1b4"}
Oct 30 19:49:47.344593 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:47.344134    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm" event={"ID":"fd0d4ac0-27c1-4735-b333-63b4e75afe70","Type":"ContainerStarted","Data":"f08ada36b720a6941ccb3e8594a40f3cc5dfe8e2df5876cbf3657ce2a5ca8b1f"}
Oct 30 19:49:47.344593 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:47.344165    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"
Oct 30 19:49:47.404425 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:47.404371    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm" podStartSLOduration=1.40435763 podStartE2EDuration="1.40435763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:47.402566878 +0000 UTC m=+1187.295181202" watchObservedRunningTime="2026-10-30 19:49:47.40435763 +0000 UTC m=+1187.296971953"
Oct 30 19:49:48.280891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.280840    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"]
Oct 30 19:49:48.283900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.283881    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"
Oct 30 19:49:48.299286 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.299256    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-5f933e\""
Oct 30 19:49:48.300238 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.300214    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-5f933e-dockercfg-st94l\""
Oct 30 19:49:48.304522 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.304500    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"]
Oct 30 19:49:48.307052 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.307038    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"
Oct 30 19:49:48.328115 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.328084    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/815c2cbd-47e1-49ef-83cf-3f96ef171d74-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v\" (UID: \"815c2cbd-47e1-49ef-83cf-3f96ef171d74\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"
Oct 30 19:49:48.328239 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.328179    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvpb4\" (UniqueName: \"kubernetes.io/projected/815c2cbd-47e1-49ef-83cf-3f96ef171d74-kube-api-access-cvpb4\") pod \"payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v\" (UID: \"815c2cbd-47e1-49ef-83cf-3f96ef171d74\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"
Oct 30 19:49:48.328398 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.328365    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8p66\" (UniqueName: \"kubernetes.io/projected/00c823a0-8617-4d7c-8df8-8b69ff439424-kube-api-access-n8p66\") pod \"payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99\" (UID: \"00c823a0-8617-4d7c-8df8-8b69ff439424\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"
Oct 30 19:49:48.328510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.328419    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c823a0-8617-4d7c-8df8-8b69ff439424-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99\" (UID: \"00c823a0-8617-4d7c-8df8-8b69ff439424\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"
Oct 30 19:49:48.341235 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.341198    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"]
Oct 30 19:49:48.385288 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.385260    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"]
Oct 30 19:49:48.429451 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.429422    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/815c2cbd-47e1-49ef-83cf-3f96ef171d74-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v\" (UID: \"815c2cbd-47e1-49ef-83cf-3f96ef171d74\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"
Oct 30 19:49:48.429451 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.429455    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvpb4\" (UniqueName: \"kubernetes.io/projected/815c2cbd-47e1-49ef-83cf-3f96ef171d74-kube-api-access-cvpb4\") pod \"payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v\" (UID: \"815c2cbd-47e1-49ef-83cf-3f96ef171d74\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"
Oct 30 19:49:48.429707 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.429516    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8p66\" (UniqueName: \"kubernetes.io/projected/00c823a0-8617-4d7c-8df8-8b69ff439424-kube-api-access-n8p66\") pod \"payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99\" (UID: \"00c823a0-8617-4d7c-8df8-8b69ff439424\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"
Oct 30 19:49:48.429707 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.429542    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c823a0-8617-4d7c-8df8-8b69ff439424-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99\" (UID: \"00c823a0-8617-4d7c-8df8-8b69ff439424\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"
Oct 30 19:49:48.430227 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.430205    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/815c2cbd-47e1-49ef-83cf-3f96ef171d74-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v\" (UID: \"815c2cbd-47e1-49ef-83cf-3f96ef171d74\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"
Oct 30 19:49:48.430372 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.430352    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c823a0-8617-4d7c-8df8-8b69ff439424-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99\" (UID: \"00c823a0-8617-4d7c-8df8-8b69ff439424\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"
Oct 30 19:49:48.471486 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.471455    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8p66\" (UniqueName: \"kubernetes.io/projected/00c823a0-8617-4d7c-8df8-8b69ff439424-kube-api-access-n8p66\") pod \"payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99\" (UID: \"00c823a0-8617-4d7c-8df8-8b69ff439424\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"
Oct 30 19:49:48.475750 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.475697    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvpb4\" (UniqueName: \"kubernetes.io/projected/815c2cbd-47e1-49ef-83cf-3f96ef171d74-kube-api-access-cvpb4\") pod \"payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v\" (UID: \"815c2cbd-47e1-49ef-83cf-3f96ef171d74\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"
Oct 30 19:49:48.601712 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.601640    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"
Oct 30 19:49:48.618951 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.618918    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"
Oct 30 19:49:48.907431 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.907393    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"]
Oct 30 19:49:48.912372 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:49:48.912342    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815c2cbd_47e1_49ef_83cf_3f96ef171d74.slice/crio-d8e2d5cb7711c1d8e8edcb01166e1da295944f1402b8c2ad445d7b992cd9ccc8 WatchSource:0}: Error finding container d8e2d5cb7711c1d8e8edcb01166e1da295944f1402b8c2ad445d7b992cd9ccc8: Status 404 returned error can't find the container with id d8e2d5cb7711c1d8e8edcb01166e1da295944f1402b8c2ad445d7b992cd9ccc8
Oct 30 19:49:48.934117 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.934088    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:49:48.960358 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:48.960134    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"]
Oct 30 19:49:48.960610 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:49:48.960577    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c823a0_8617_4d7c_8df8_8b69ff439424.slice/crio-122dcec6a637b8ec1aa2ec3fb65a4fec793867d6f2f26e7436bc347276a0982a WatchSource:0}: Error finding container 122dcec6a637b8ec1aa2ec3fb65a4fec793867d6f2f26e7436bc347276a0982a: Status 404 returned error can't find the container with id 122dcec6a637b8ec1aa2ec3fb65a4fec793867d6f2f26e7436bc347276a0982a
Oct 30 19:49:49.155322 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.155288    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"]
Oct 30 19:49:49.158339 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.158285    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"
Oct 30 19:49:49.170949 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.170930    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-8a8207\""
Oct 30 19:49:49.177845 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.177563    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"]
Oct 30 19:49:49.219479 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.219447    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"]
Oct 30 19:49:49.222526 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.222503    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"
Oct 30 19:49:49.228780 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.228757    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-8a8207-dockercfg-grhw8\""
Oct 30 19:49:49.244106 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.244077    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"]
Oct 30 19:49:49.249569 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.249540    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8f8f91-8f8b-4f8b-a0d9-3483580a7855-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p\" (UID: \"ca8f8f91-8f8b-4f8b-a0d9-3483580a7855\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"
Oct 30 19:49:49.249715 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.249692    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb7a21b-3c5f-446b-81ee-732307718f1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9\" (UID: \"dbb7a21b-3c5f-446b-81ee-732307718f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"
Oct 30 19:49:49.249839 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.249813    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68g6x\" (UniqueName: \"kubernetes.io/projected/ca8f8f91-8f8b-4f8b-a0d9-3483580a7855-kube-api-access-68g6x\") pod \"payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p\" (UID: \"ca8f8f91-8f8b-4f8b-a0d9-3483580a7855\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"
Oct 30 19:49:49.249939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.249921    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nvv9\" (UniqueName: \"kubernetes.io/projected/dbb7a21b-3c5f-446b-81ee-732307718f1b-kube-api-access-7nvv9\") pod \"payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9\" (UID: \"dbb7a21b-3c5f-446b-81ee-732307718f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"
Oct 30 19:49:49.350965 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.350922    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nvv9\" (UniqueName: \"kubernetes.io/projected/dbb7a21b-3c5f-446b-81ee-732307718f1b-kube-api-access-7nvv9\") pod \"payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9\" (UID: \"dbb7a21b-3c5f-446b-81ee-732307718f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"
Oct 30 19:49:49.351153 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.351065    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8f8f91-8f8b-4f8b-a0d9-3483580a7855-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p\" (UID: \"ca8f8f91-8f8b-4f8b-a0d9-3483580a7855\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"
Oct 30 19:49:49.351218 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.351182    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb7a21b-3c5f-446b-81ee-732307718f1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9\" (UID: \"dbb7a21b-3c5f-446b-81ee-732307718f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"
Oct 30 19:49:49.351481 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.351437    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68g6x\" (UniqueName: \"kubernetes.io/projected/ca8f8f91-8f8b-4f8b-a0d9-3483580a7855-kube-api-access-68g6x\") pod \"payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p\" (UID: \"ca8f8f91-8f8b-4f8b-a0d9-3483580a7855\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"
Oct 30 19:49:49.352023 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.352000    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8f8f91-8f8b-4f8b-a0d9-3483580a7855-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p\" (UID: \"ca8f8f91-8f8b-4f8b-a0d9-3483580a7855\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"
Oct 30 19:49:49.352124 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.352049    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb7a21b-3c5f-446b-81ee-732307718f1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9\" (UID: \"dbb7a21b-3c5f-446b-81ee-732307718f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"
Oct 30 19:49:49.357560 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.357533    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v" event={"ID":"815c2cbd-47e1-49ef-83cf-3f96ef171d74","Type":"ContainerStarted","Data":"11dac1c7475d86cb0f4a2909b6bd2b55fc972a22dfcdac54473a91f7cddc6f11"}
Oct 30 19:49:49.357686 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.357599    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v" event={"ID":"815c2cbd-47e1-49ef-83cf-3f96ef171d74","Type":"ContainerStarted","Data":"d8e2d5cb7711c1d8e8edcb01166e1da295944f1402b8c2ad445d7b992cd9ccc8"}
Oct 30 19:49:49.357686 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.357632    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"
Oct 30 19:49:49.359053 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.359032    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99" event={"ID":"00c823a0-8617-4d7c-8df8-8b69ff439424","Type":"ContainerStarted","Data":"e4d37713a969f0bd3edd2ab863c57374fb07ca0ea85e01dc6fcc9862f356617d"}
Oct 30 19:49:49.359146 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.359075    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99" event={"ID":"00c823a0-8617-4d7c-8df8-8b69ff439424","Type":"ContainerStarted","Data":"122dcec6a637b8ec1aa2ec3fb65a4fec793867d6f2f26e7436bc347276a0982a"}
Oct 30 19:49:49.359146 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.359095    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"
Oct 30 19:49:49.369191 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.369169    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nvv9\" (UniqueName: \"kubernetes.io/projected/dbb7a21b-3c5f-446b-81ee-732307718f1b-kube-api-access-7nvv9\") pod \"payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9\" (UID: \"dbb7a21b-3c5f-446b-81ee-732307718f1b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"
Oct 30 19:49:49.369280 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.369216    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68g6x\" (UniqueName: \"kubernetes.io/projected/ca8f8f91-8f8b-4f8b-a0d9-3483580a7855-kube-api-access-68g6x\") pod \"payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p\" (UID: \"ca8f8f91-8f8b-4f8b-a0d9-3483580a7855\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"
Oct 30 19:49:49.382684 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.382641    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v" podStartSLOduration=1.3826293029999999 podStartE2EDuration="1.382629303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:49.380509418 +0000 UTC m=+1189.273123741" watchObservedRunningTime="2026-10-30 19:49:49.382629303 +0000 UTC m=+1189.275243626"
Oct 30 19:49:49.470875 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.470572    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"
Oct 30 19:49:49.538092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.536523    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"
Oct 30 19:49:49.765934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.765823    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99" podStartSLOduration=1.7658059179999999 podStartE2EDuration="1.765805918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:49.422901895 +0000 UTC m=+1189.315516219" watchObservedRunningTime="2026-10-30 19:49:49.765805918 +0000 UTC m=+1189.658420246"
Oct 30 19:49:49.767593 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.767572    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"]
Oct 30 19:49:49.770151 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:49:49.770119    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb7a21b_3c5f_446b_81ee_732307718f1b.slice/crio-ac1d839ea6831fed4fa4e51e4de05d96626c1ef2306f31602741a04269e44082 WatchSource:0}: Error finding container ac1d839ea6831fed4fa4e51e4de05d96626c1ef2306f31602741a04269e44082: Status 404 returned error can't find the container with id ac1d839ea6831fed4fa4e51e4de05d96626c1ef2306f31602741a04269e44082
Oct 30 19:49:49.843409 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:49.843384    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"]
Oct 30 19:49:49.845452 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:49:49.845429    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8f8f91_8f8b_4f8b_a0d9_3483580a7855.slice/crio-64a9320ecb12f1c6310b80741f99423689aaee97a72465b8932e788e3614c1f6 WatchSource:0}: Error finding container 64a9320ecb12f1c6310b80741f99423689aaee97a72465b8932e788e3614c1f6: Status 404 returned error can't find the container with id 64a9320ecb12f1c6310b80741f99423689aaee97a72465b8932e788e3614c1f6
Oct 30 19:49:50.382009 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:50.381965    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9" event={"ID":"dbb7a21b-3c5f-446b-81ee-732307718f1b","Type":"ContainerStarted","Data":"4259dfbbda23a228d18f5b93f38a56bfc37de94c8167fd8b77b4e4288262c58e"}
Oct 30 19:49:50.382220 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:50.382030    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"
Oct 30 19:49:50.382220 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:50.382055    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9" event={"ID":"dbb7a21b-3c5f-446b-81ee-732307718f1b","Type":"ContainerStarted","Data":"ac1d839ea6831fed4fa4e51e4de05d96626c1ef2306f31602741a04269e44082"}
Oct 30 19:49:50.383596 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:50.383569    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p" event={"ID":"ca8f8f91-8f8b-4f8b-a0d9-3483580a7855","Type":"ContainerStarted","Data":"a0e76d316f44f8576c6a86d71e222be98729fe82c4ff45ec8419f7a11b430c53"}
Oct 30 19:49:50.383718 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:50.383617    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p" event={"ID":"ca8f8f91-8f8b-4f8b-a0d9-3483580a7855","Type":"ContainerStarted","Data":"64a9320ecb12f1c6310b80741f99423689aaee97a72465b8932e788e3614c1f6"}
Oct 30 19:49:50.383899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:50.383852    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"
Oct 30 19:49:50.415051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:50.415000    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9" podStartSLOduration=1.414986187 podStartE2EDuration="1.414986187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:50.41332569 +0000 UTC m=+1190.305940013" watchObservedRunningTime="2026-10-30 19:49:50.414986187 +0000 UTC m=+1190.307600514"
Oct 30 19:49:50.437801 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:50.437740    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p" podStartSLOduration=1.43772538 podStartE2EDuration="1.43772538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:50.435210005 +0000 UTC m=+1190.327824329" watchObservedRunningTime="2026-10-30 19:49:50.43772538 +0000 UTC m=+1190.330339703"
Oct 30 19:49:53.117967 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:53.117932    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:49:58.327025 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:58.326989    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:49:58.327416 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:58.327054    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws" podUID="091b9ee1-723f-44d7-bac5-629178dbd234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:58.377145 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:58.377102    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"
Oct 30 19:49:59.130365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:49:59.130330    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:50:00.428886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:00.418419    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"
Oct 30 19:50:00.430776 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:00.430750    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"
Oct 30 19:50:00.540611 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:00.540521    4823 generic.go:356] "Generic (PLEG): container finished" podID="091b9ee1-723f-44d7-bac5-629178dbd234" containerID="16c71fad5a5ad82b8bf869736e059b942aff2723a3249185755840d3110a9b79" exitCode=0
Oct 30 19:50:00.540851 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:00.540558    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws" event={"ID":"091b9ee1-723f-44d7-bac5-629178dbd234","Type":"ContainerDied","Data":"16c71fad5a5ad82b8bf869736e059b942aff2723a3249185755840d3110a9b79"}
Oct 30 19:50:00.541256 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:00.541237    4823 scope.go:122] "RemoveContainer" containerID="16c71fad5a5ad82b8bf869736e059b942aff2723a3249185755840d3110a9b79"
Oct 30 19:50:00.543461 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:00.543215    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 19:50:00.543615 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:00.543590    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:50:00.543768 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:00.543741    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:50:00.543901 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:00.543888    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:50:00.545317 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:00.545016    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 19:50:01.408905 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:01.407987    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"
Oct 30 19:50:01.409238 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:01.409216    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"
Oct 30 19:50:01.586889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:01.582666    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws" event={"ID":"091b9ee1-723f-44d7-bac5-629178dbd234","Type":"ContainerStarted","Data":"a8db6bb9b415b6b60261828b6ac44c80194095b9c245aed2712caf1acee58521"}
Oct 30 19:50:01.860898 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:01.860254    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:50:01.860898 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:01.860383    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:50:01.860898 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:01.860429    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws" podUID="091b9ee1-723f-44d7-bac5-629178dbd234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:50:02.048920 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:50:02.048884    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:50:02.859884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:02.859362    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:50:02.859884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:02.859457    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws" podUID="091b9ee1-723f-44d7-bac5-629178dbd234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:50:03.863423 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:03.863387    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:50:03.864178 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:03.864141    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws" podUID="091b9ee1-723f-44d7-bac5-629178dbd234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:50:04.860882 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:04.860361    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:50:04.860882 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:04.860438    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws" podUID="091b9ee1-723f-44d7-bac5-629178dbd234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:50:05.871889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:05.869243    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:50:05.871889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:05.869306    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws" podUID="091b9ee1-723f-44d7-bac5-629178dbd234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:50:05.894888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:05.880117    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:50:06.822656 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:06.819951    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"]
Oct 30 19:50:06.828374 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:06.828347    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"]
Oct 30 19:50:06.828569 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:06.828465    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:06.835399 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:06.835252    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"openshift-service-ca.crt\""
Oct 30 19:50:06.835399 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:06.835371    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"kube-root-ca.crt\""
Oct 30 19:50:06.846085 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:06.844429    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"default-dockercfg-6842t\""
Oct 30 19:50:06.846085 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:06.844566    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"simulator-w4-3f57c5-kserve-self-signed-certs\""
Oct 30 19:50:06.887959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:06.887527    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:50:06.887959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:06.887584    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws" podUID="091b9ee1-723f-44d7-bac5-629178dbd234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:50:06.934993 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:06.934958    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t9tz\" (UniqueName: \"kubernetes.io/projected/60e95f14-dd0c-47d5-92ef-5acc29210a5d-kube-api-access-8t9tz\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:06.934993 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:06.934998    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-home\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:06.935240 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:06.935064    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-model-cache\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:06.935240 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:06.935083    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e95f14-dd0c-47d5-92ef-5acc29210a5d-tls-certs\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:06.935240 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:06.935116    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-dshm\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:06.935240 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:06.935143    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-tmp-dir\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:07.037895 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.036618    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t9tz\" (UniqueName: \"kubernetes.io/projected/60e95f14-dd0c-47d5-92ef-5acc29210a5d-kube-api-access-8t9tz\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:07.037895 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.036695    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-home\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:07.037895 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.036822    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-model-cache\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:07.037895 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.036853    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e95f14-dd0c-47d5-92ef-5acc29210a5d-tls-certs\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:07.037895 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.036929    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-dshm\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:07.037895 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.036990    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-tmp-dir\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:07.037895 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.037722    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-model-cache\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:07.038545 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.038003    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-tmp-dir\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:07.038644 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.038559    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-home\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:07.041351 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.041319    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-dshm\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:07.044778 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.044751    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e95f14-dd0c-47d5-92ef-5acc29210a5d-tls-certs\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:07.095958 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.061841    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t9tz\" (UniqueName: \"kubernetes.io/projected/60e95f14-dd0c-47d5-92ef-5acc29210a5d-kube-api-access-8t9tz\") pod \"simulator-w4-3f57c5-kserve-7797496b8-4k7h6\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") " pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:07.172314 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.172277    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:07.497606 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.497573    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"]
Oct 30 19:50:07.504395 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:50:07.504344    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60e95f14_dd0c_47d5_92ef_5acc29210a5d.slice/crio-21cf894df2e6d2f70b09edbda26ec62603ec2adc01a967092dcfb03330e9cda3 WatchSource:0}: Error finding container 21cf894df2e6d2f70b09edbda26ec62603ec2adc01a967092dcfb03330e9cda3: Status 404 returned error can't find the container with id 21cf894df2e6d2f70b09edbda26ec62603ec2adc01a967092dcfb03330e9cda3
Oct 30 19:50:07.702531 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.702488    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" event={"ID":"60e95f14-dd0c-47d5-92ef-5acc29210a5d","Type":"ContainerStarted","Data":"ce2ef19d9aa332e404fcc4ba4521dac247be996219436aeb0a2fabe7169c09b1"}
Oct 30 19:50:07.702719 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.702550    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" event={"ID":"60e95f14-dd0c-47d5-92ef-5acc29210a5d","Type":"ContainerStarted","Data":"21cf894df2e6d2f70b09edbda26ec62603ec2adc01a967092dcfb03330e9cda3"}
Oct 30 19:50:07.737360 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:07.737286    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podStartSLOduration=1.7372659339999998 podStartE2EDuration="1.737265934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:07.72882955 +0000 UTC m=+1207.621443876" watchObservedRunningTime="2026-10-30 19:50:07.737265934 +0000 UTC m=+1207.629880260"
Oct 30 19:50:08.170378 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.170346    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:50:08.171125 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.171104    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:50:08.171878 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.171835    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 19:50:08.172946 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.172927    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:08.250721 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.250634    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"]
Oct 30 19:50:08.258233 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.258201    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.270402 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.268138    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"simulator-w3-8a8207-kserve-self-signed-certs\""
Oct 30 19:50:08.270402 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.268273    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"default-dockercfg-lb6sq\""
Oct 30 19:50:08.296280 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.296249    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"]
Oct 30 19:50:08.360843 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.360818    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"openshift-service-ca.crt\""
Oct 30 19:50:08.361084 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.361060    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"kube-root-ca.crt\""
Oct 30 19:50:08.370638 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.370600    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-home\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.370816 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.370654    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6pn8\" (UniqueName: \"kubernetes.io/projected/499b7351-9dea-47b8-8e88-a532f7fab38f-kube-api-access-w6pn8\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.370816 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.370800    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-model-cache\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.370941 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.370837    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/499b7351-9dea-47b8-8e88-a532f7fab38f-tls-certs\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.371008 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.370943    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-tmp-dir\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.371008 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.370992    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-dshm\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.472645 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.472542    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-tmp-dir\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.472645 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.472614    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-dshm\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.472952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.472706    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-home\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.472952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.472745    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6pn8\" (UniqueName: \"kubernetes.io/projected/499b7351-9dea-47b8-8e88-a532f7fab38f-kube-api-access-w6pn8\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.472952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.472790    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-model-cache\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.472952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.472836    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/499b7351-9dea-47b8-8e88-a532f7fab38f-tls-certs\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.473915 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.473659    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-home\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.473915 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.473805    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-tmp-dir\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.474106 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.473939    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-model-cache\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.477519 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.476131    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/499b7351-9dea-47b8-8e88-a532f7fab38f-tls-certs\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.481228 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.481140    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-dshm\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.487022 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.486990    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6pn8\" (UniqueName: \"kubernetes.io/projected/499b7351-9dea-47b8-8e88-a532f7fab38f-kube-api-access-w6pn8\") pod \"simulator-w3-8a8207-kserve-864587d478-kj8bz\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") " pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.588667 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.588630    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:08.899913 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:08.899829    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"]
Oct 30 19:50:09.722095 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:09.722055    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" event={"ID":"499b7351-9dea-47b8-8e88-a532f7fab38f","Type":"ContainerStarted","Data":"ead3662d68a6479c3abc9c4acc79c953001ed92f9f81e79e738ee9bf75d42536"}
Oct 30 19:50:09.722632 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:09.722110    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" event={"ID":"499b7351-9dea-47b8-8e88-a532f7fab38f","Type":"ContainerStarted","Data":"54c85b2ba71fd5c4eee9bf0bef0db6213e35be9682cb049d9dbf3036f11c281c"}
Oct 30 19:50:09.743241 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:09.743182    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podStartSLOduration=1.7431685 podStartE2EDuration="1.7431685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:09.740285936 +0000 UTC m=+1209.632900265" watchObservedRunningTime="2026-10-30 19:50:09.7431685 +0000 UTC m=+1209.635782824"
Oct 30 19:50:10.162953 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.129779    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"]
Oct 30 19:50:10.162953 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.147799    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"]
Oct 30 19:50:10.162953 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.147956    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.162953 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.159114    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"openshift-service-ca.crt\""
Oct 30 19:50:10.162953 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.159294    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"kube-root-ca.crt\""
Oct 30 19:50:10.162953 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.159385    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"simulator-w1-a1d885-kserve-self-signed-certs\""
Oct 30 19:50:10.162953 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.159551    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"default-dockercfg-fzr2x\""
Oct 30 19:50:10.193012 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.192934    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-model-cache\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.193169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.193121    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-home\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.193222 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.193204    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-dshm\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.193271 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.193257    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-tmp-dir\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.193384 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.193361    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxqw7\" (UniqueName: \"kubernetes.io/projected/dd230496-5fbe-4500-965c-16e48f099552-kube-api-access-kxqw7\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.193457 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.193442    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd230496-5fbe-4500-965c-16e48f099552-tls-certs\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.294633 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.294595    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-model-cache\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.294811 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.294732    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-home\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.294811 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.294801    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-dshm\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.294926 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.294846    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-tmp-dir\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.294926 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.294917    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxqw7\" (UniqueName: \"kubernetes.io/projected/dd230496-5fbe-4500-965c-16e48f099552-kube-api-access-kxqw7\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.295034 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.294957    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd230496-5fbe-4500-965c-16e48f099552-tls-certs\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.295034 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.294977    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-model-cache\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.295172 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.295142    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-home\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.295285 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.295255    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-tmp-dir\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.297621 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.297597    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-dshm\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.300633 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.300608    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd230496-5fbe-4500-965c-16e48f099552-tls-certs\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.314978 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.313887    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxqw7\" (UniqueName: \"kubernetes.io/projected/dd230496-5fbe-4500-965c-16e48f099552-kube-api-access-kxqw7\") pod \"simulator-w1-a1d885-kserve-6794f9f86c-ldhqk\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") " pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.465315 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.465227    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:10.589230 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.589199    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:10.757373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.735092    4823 generic.go:356] "Generic (PLEG): container finished" podID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" containerID="c9ff1fae34a53e9dd981298a9664a6cf4fa8b421897d3b40e80b50625cd56853" exitCode=0
Oct 30 19:50:10.757373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.735156    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd" event={"ID":"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0","Type":"ContainerDied","Data":"c9ff1fae34a53e9dd981298a9664a6cf4fa8b421897d3b40e80b50625cd56853"}
Oct 30 19:50:10.757373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.735641    4823 scope.go:122] "RemoveContainer" containerID="c9ff1fae34a53e9dd981298a9664a6cf4fa8b421897d3b40e80b50625cd56853"
Oct 30 19:50:10.757373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.737037    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:50:10.757373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.737115    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:50:10.757373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.737160    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:50:10.914550 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:10.914450    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"]
Oct 30 19:50:11.742057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:11.742013    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" event={"ID":"dd230496-5fbe-4500-965c-16e48f099552","Type":"ContainerStarted","Data":"966de27847ce9e4d0f3da21d8abca77d96a8d6d3c569c23b7625693b17d2f22c"}
Oct 30 19:50:11.742057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:11.742059    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" event={"ID":"dd230496-5fbe-4500-965c-16e48f099552","Type":"ContainerStarted","Data":"df3e62e60a4cda7c93f3240e305f8e8aa950113931485146a4e6090fc2426f81"}
Oct 30 19:50:11.743729 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:11.743702    4823 generic.go:356] "Generic (PLEG): container finished" podID="82f08656-55bf-4de5-9beb-3776bc8dd3f2" containerID="2208641275ce75519737c1cd9455c82a180c032775afbfcf7b62c1002c93a7e6" exitCode=0
Oct 30 19:50:11.743845 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:11.743771    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm" event={"ID":"82f08656-55bf-4de5-9beb-3776bc8dd3f2","Type":"ContainerDied","Data":"2208641275ce75519737c1cd9455c82a180c032775afbfcf7b62c1002c93a7e6"}
Oct 30 19:50:11.744189 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:11.744172    4823 scope.go:122] "RemoveContainer" containerID="2208641275ce75519737c1cd9455c82a180c032775afbfcf7b62c1002c93a7e6"
Oct 30 19:50:11.745577 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:11.745547    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:50:11.745684 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:11.745621    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:50:11.745684 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:11.745655    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:50:11.746019 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:11.745997    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd" event={"ID":"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0","Type":"ContainerStarted","Data":"32c425d21eb4fa1e252b330c4f0cc54e85f51727ae1fb31d13cbf5595da88b46"}
Oct 30 19:50:11.747677 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:11.747658    4823 generic.go:356] "Generic (PLEG): container finished" podID="dd984f9d-dcce-452e-a843-e07894b05c69" containerID="91ce26b56cb6dc68cd1b497564dafeac233a8113118979450423f525660b5e3c" exitCode=0
Oct 30 19:50:11.747798 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:11.747736    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44" event={"ID":"dd984f9d-dcce-452e-a843-e07894b05c69","Type":"ContainerDied","Data":"91ce26b56cb6dc68cd1b497564dafeac233a8113118979450423f525660b5e3c"}
Oct 30 19:50:11.748083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:11.748067    4823 scope.go:122] "RemoveContainer" containerID="91ce26b56cb6dc68cd1b497564dafeac233a8113118979450423f525660b5e3c"
Oct 30 19:50:11.749150 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:11.749125    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:50:11.749232 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:11.749205    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:50:11.749281 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:11.749238    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:50:11.807342 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:11.807295    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podStartSLOduration=1.807280658 podStartE2EDuration="1.807280658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:11.767077892 +0000 UTC m=+1211.659692218" watchObservedRunningTime="2026-10-30 19:50:11.807280658 +0000 UTC m=+1211.699895027"
Oct 30 19:50:12.094753 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:12.094651    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:50:12.094980 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:12.094806    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:50:12.094980 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:12.094880    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd" podUID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 19:50:12.284726 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:12.284688    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:50:12.470886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:12.468429    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:12.807946 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:12.800585    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44" event={"ID":"dd984f9d-dcce-452e-a843-e07894b05c69","Type":"ContainerStarted","Data":"0da710fce960157285e7ccb5eb1697e859a7c1f31b0cb06eb0c9ab956c20719e"}
Oct 30 19:50:12.833793 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:12.832950    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm" event={"ID":"82f08656-55bf-4de5-9beb-3776bc8dd3f2","Type":"ContainerStarted","Data":"dfad4186f2aeae3d8c24ddf6a94b9c5999592637abc0be617df6e7f3a0ed4a02"}
Oct 30 19:50:13.101940 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:13.101366    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:50:13.101940 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:13.101432    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd" podUID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 19:50:13.266896 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:13.264262    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:50:13.266896 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:13.264411    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:50:13.266896 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:13.264454    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44" podUID="dd984f9d-dcce-452e-a843-e07894b05c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 19:50:13.299886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:13.298977    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:50:13.300100 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:13.299919    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:50:13.594893 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:13.594518    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:50:13.604057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:13.604026    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:50:13.604350 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:13.604304    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm" podUID="82f08656-55bf-4de5-9beb-3776bc8dd3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 19:50:14.098887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:14.098357    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:50:14.098887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:14.098420    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd" podUID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 19:50:14.257234 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:14.257201    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:50:14.257520 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:14.257483    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44" podUID="dd984f9d-dcce-452e-a843-e07894b05c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 19:50:14.584227 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:14.584190    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:50:14.584509 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:14.584478    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm" podUID="82f08656-55bf-4de5-9beb-3776bc8dd3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 19:50:15.095314 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:15.095271    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:50:15.095631 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:15.095598    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd" podUID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 19:50:15.266070 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:15.266030    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:50:15.266689 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:15.266653    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44" podUID="dd984f9d-dcce-452e-a843-e07894b05c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 19:50:15.593887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:15.592318    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:50:15.593887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:15.592395    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm" podUID="82f08656-55bf-4de5-9beb-3776bc8dd3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 19:50:16.098886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:16.098428    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:50:16.098886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:16.098492    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd" podUID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 19:50:16.264887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:16.262260    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:50:16.264887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:16.262323    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44" podUID="dd984f9d-dcce-452e-a843-e07894b05c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 19:50:16.583887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:16.583408    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:50:16.583887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:16.583489    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm" podUID="82f08656-55bf-4de5-9beb-3776bc8dd3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 19:50:17.103887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:17.099264    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:50:17.103887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:17.099326    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd" podUID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 19:50:17.178027 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:17.177992    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:17.220648 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:17.220621    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:17.261896 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:17.261126    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:50:17.261896 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:17.261183    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44" podUID="dd984f9d-dcce-452e-a843-e07894b05c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 30 19:50:17.583281 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:17.583242    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:50:17.583590 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:17.583556    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm" podUID="82f08656-55bf-4de5-9beb-3776bc8dd3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 30 19:50:17.950694 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:17.950664    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:50:18.098035 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:18.098002    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:50:18.099436 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:18.099400    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 19:50:18.314275 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:18.314181    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:50:18.314615 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:18.314584    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:50:18.315654 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:18.315631    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 19:50:18.587649 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:18.587579    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:50:18.588612 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:18.588592    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 19:50:18.589705 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:18.589693    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:18.601851 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:18.601829    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:18.957155 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:18.957124    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:50:19.118278 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:19.118236    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:50:20.465941 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:20.465904    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:20.478296 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:20.478267    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:20.971355 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:20.971327    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:50:32.013569 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:50:32.013533    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:50:33.114021 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.113981    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:50:33.248467 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.248432    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"]
Oct 30 19:50:33.253751 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.253728    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.263506 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.263476    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"premium-w3-8a8207-kserve-self-signed-certs\""
Oct 30 19:50:33.263776 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.263755    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"]
Oct 30 19:50:33.351506 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.351466    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-dshm\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.351689 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.351511    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcdm2\" (UniqueName: \"kubernetes.io/projected/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-kube-api-access-jcdm2\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.351689 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.351536    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-home\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.351689 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.351682    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-tls-certs\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.351827 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.351728    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-model-cache\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.351907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.351847    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-tmp-dir\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.453783 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.453680    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-tmp-dir\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.454170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.453855    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-dshm\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.454170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.453923    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcdm2\" (UniqueName: \"kubernetes.io/projected/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-kube-api-access-jcdm2\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.454170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.453956    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-home\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.454170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.454045    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-tls-certs\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.454170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.454140    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-model-cache\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.454170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.454155    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-tmp-dir\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.456077 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.454727    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-home\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.488406 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.477172    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-model-cache\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.497330 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.497283    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-dshm\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.497612 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.497587    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-tls-certs\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.497982 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.497914    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcdm2\" (UniqueName: \"kubernetes.io/projected/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-kube-api-access-jcdm2\") pod \"premium-w3-8a8207-kserve-94d6f88cf-mlnkj\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") " pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.568373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.568334    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:33.881686 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:33.881654    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"]
Oct 30 19:50:33.883255 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:50:33.883224    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62c3dcdc_3ec3_4a36_b7ac_2543ff2c1718.slice/crio-9de131df7143f54b41efc2add13f3ee71388f9e9654b9ae9e3b3922828e38ea3 WatchSource:0}: Error finding container 9de131df7143f54b41efc2add13f3ee71388f9e9654b9ae9e3b3922828e38ea3: Status 404 returned error can't find the container with id 9de131df7143f54b41efc2add13f3ee71388f9e9654b9ae9e3b3922828e38ea3
Oct 30 19:50:34.038682 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:34.038642    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" event={"ID":"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718","Type":"ContainerStarted","Data":"520da7347061651d9fe736ebaa3048520d02f137c2332420e130dcce2e108335"}
Oct 30 19:50:34.038894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:34.038695    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" event={"ID":"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718","Type":"ContainerStarted","Data":"9de131df7143f54b41efc2add13f3ee71388f9e9654b9ae9e3b3922828e38ea3"}
Oct 30 19:50:34.058708 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:34.058661    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podStartSLOduration=1.058648337 podStartE2EDuration="1.058648337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:34.057189559 +0000 UTC m=+1233.949803883" watchObservedRunningTime="2026-10-30 19:50:34.058648337 +0000 UTC m=+1233.951262660"
Oct 30 19:50:34.568625 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:34.568589    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:42.051148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.051117    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"]
Oct 30 19:50:42.055412 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.055385    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.063889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.063844    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ced832\"/\"kube-root-ca.crt\""
Oct 30 19:50:42.064029 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.063907    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ced832\"/\"openshift-service-ca.crt\""
Oct 30 19:50:42.064029 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.063944    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ced832\"/\"premium-w0-ced832-kserve-self-signed-certs\""
Oct 30 19:50:42.064147 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.064127    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ced832\"/\"default-dockercfg-vjsbn\""
Oct 30 19:50:42.073591 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.073568    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"]
Oct 30 19:50:42.168070 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.168034    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-model-cache\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.168283 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.168095    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svqnf\" (UniqueName: \"kubernetes.io/projected/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-kube-api-access-svqnf\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.168283 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.168218    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-tls-certs\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.168412 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.168324    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-dshm\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.168412 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.168345    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-tmp-dir\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.168524 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.168471    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-home\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.269150 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.269111    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-home\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.269307 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.269170    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-model-cache\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.269307 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.269209    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svqnf\" (UniqueName: \"kubernetes.io/projected/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-kube-api-access-svqnf\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.269307 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.269238    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-tls-certs\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.269497 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.269320    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-dshm\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.269497 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.269341    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-tmp-dir\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.269666 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.269643    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-model-cache\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.269719 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.269671    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-home\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.269756 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.269730    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-tmp-dir\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.271673 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.271645    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-dshm\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.271971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.271945    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-tls-certs\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.280979 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.280958    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svqnf\" (UniqueName: \"kubernetes.io/projected/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-kube-api-access-svqnf\") pod \"premium-w0-ced832-kserve-59d489cd49-7xmmk\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") " pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.368399 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.368359    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:42.651954 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:42.651929    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"]
Oct 30 19:50:42.654271 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:50:42.654246    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a2ff89f_2dce_4e68_a8b8_443581fb78ee.slice/crio-5f78420c1333735e064ec3ce3461ef6f70bf1b9b9a2e7311098a113475e123d7 WatchSource:0}: Error finding container 5f78420c1333735e064ec3ce3461ef6f70bf1b9b9a2e7311098a113475e123d7: Status 404 returned error can't find the container with id 5f78420c1333735e064ec3ce3461ef6f70bf1b9b9a2e7311098a113475e123d7
Oct 30 19:50:43.099297 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:43.099202    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" event={"ID":"6a2ff89f-2dce-4e68-a8b8-443581fb78ee","Type":"ContainerStarted","Data":"10974dbd2acbb0f904f1bf2f5b69e10228b7e498a7681377c119168bc30e4277"}
Oct 30 19:50:43.099297 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:43.099248    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" event={"ID":"6a2ff89f-2dce-4e68-a8b8-443581fb78ee","Type":"ContainerStarted","Data":"5f78420c1333735e064ec3ce3461ef6f70bf1b9b9a2e7311098a113475e123d7"}
Oct 30 19:50:43.126942 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:43.126895    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podStartSLOduration=1.12688136 podStartE2EDuration="1.12688136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:43.125287946 +0000 UTC m=+1243.017902269" watchObservedRunningTime="2026-10-30 19:50:43.12688136 +0000 UTC m=+1243.019495678"
Oct 30 19:50:43.368629 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:43.368597    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:43.568988 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:43.568951    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:43.581649 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:43.581625    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:44.116161 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:44.116134    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:50:48.399658 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.399622    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"]
Oct 30 19:50:48.403714 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.403692    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.412856 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.412831    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"openshift-service-ca.crt\""
Oct 30 19:50:48.413140 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.413119    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"default-dockercfg-dsn78\""
Oct 30 19:50:48.413235 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.413221    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"kube-root-ca.crt\""
Oct 30 19:50:48.413295 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.413275    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"premium-w6-5f933e-kserve-self-signed-certs\""
Oct 30 19:50:48.416447 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.416421    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"]
Oct 30 19:50:48.545642 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.545588    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-home\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.545907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.545648    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-dshm\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.545907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.545717    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-model-cache\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.545907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.545749    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-tmp-dir\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.545907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.545877    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h5tf\" (UniqueName: \"kubernetes.io/projected/ca283777-7610-46f7-9fc2-cd13b31f7209-kube-api-access-2h5tf\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.546135 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.545955    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca283777-7610-46f7-9fc2-cd13b31f7209-tls-certs\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.647333 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.647288    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca283777-7610-46f7-9fc2-cd13b31f7209-tls-certs\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.647532 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.647389    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-home\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.647532 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.647412    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-dshm\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.647532 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.647472    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-model-cache\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.647532 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.647498    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-tmp-dir\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.647751 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.647604    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h5tf\" (UniqueName: \"kubernetes.io/projected/ca283777-7610-46f7-9fc2-cd13b31f7209-kube-api-access-2h5tf\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.647914 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.647848    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-home\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.648013 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.647963    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-model-cache\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.648013 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.648003    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-tmp-dir\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.649941 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.649899    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca283777-7610-46f7-9fc2-cd13b31f7209-tls-certs\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.650966 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.650949    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-dshm\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.659943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.659917    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h5tf\" (UniqueName: \"kubernetes.io/projected/ca283777-7610-46f7-9fc2-cd13b31f7209-kube-api-access-2h5tf\") pod \"premium-w6-5f933e-kserve-55675c5fdf-z5r26\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") " pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:48.718792 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:48.718756    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:49.017069 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:49.015946    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"]
Oct 30 19:50:49.129650 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:49.129616    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:50:49.138953 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:49.138917    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" event={"ID":"ca283777-7610-46f7-9fc2-cd13b31f7209","Type":"ContainerStarted","Data":"88defcac7b38958448c575aefe3b2501c808a79c83cdba8c1d3376ec4916005a"}
Oct 30 19:50:49.139127 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:49.138964    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" event={"ID":"ca283777-7610-46f7-9fc2-cd13b31f7209","Type":"ContainerStarted","Data":"649acbce5cf6a411210dd6d602ed733b4806aef5db95a6cdfeb86ec68d8a631b"}
Oct 30 19:50:49.159625 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:49.159571    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podStartSLOduration=1.159556211 podStartE2EDuration="1.159556211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:49.157949219 +0000 UTC m=+1249.050563543" watchObservedRunningTime="2026-10-30 19:50:49.159556211 +0000 UTC m=+1249.052170534"
Oct 30 19:50:49.719244 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:49.719208    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:52.369058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:52.369024    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:52.381754 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:52.381727    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:53.191757 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:53.191730    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:50:58.719693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:58.719653    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:58.732547 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:58.732517    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:50:59.236844 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:50:59.236815    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:51:02.013575 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:51:02.013536    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:51:06.403878 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:06.403835    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:51:09.960953 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:09.960914    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"]
Oct 30 19:51:09.966840 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:09.966818    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:09.977927 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:09.977903    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3f57c5-openshift-default-dockercfg-rshxj\""
Oct 30 19:51:09.982465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:09.982446    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"]
Oct 30 19:51:10.113663 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.113612    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/118587b4-0b24-4a51-b462-68bce9602a0a-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.113850 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.113694    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.113971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.113882    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.113971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.113924    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn26p\" (UniqueName: \"kubernetes.io/projected/118587b4-0b24-4a51-b462-68bce9602a0a-kube-api-access-jn26p\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.113971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.113956    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.114132 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.114111    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/118587b4-0b24-4a51-b462-68bce9602a0a-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.114285 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.114264    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.114368 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.114348    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/118587b4-0b24-4a51-b462-68bce9602a0a-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.114448 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.114425    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.215545 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.215449    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.215545 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.215512    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/118587b4-0b24-4a51-b462-68bce9602a0a-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.215545 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.215556    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.215820 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.215595    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/118587b4-0b24-4a51-b462-68bce9602a0a-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.215820 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.215639    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.215820 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.215676    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.215820 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.215703    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn26p\" (UniqueName: \"kubernetes.io/projected/118587b4-0b24-4a51-b462-68bce9602a0a-kube-api-access-jn26p\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.215820 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.215733    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.216192 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.215917    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.216192 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.216064    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.216192 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.216140    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.216360 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.216265    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.216514 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.216491    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/118587b4-0b24-4a51-b462-68bce9602a0a-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.217035 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.217018    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/118587b4-0b24-4a51-b462-68bce9602a0a-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.218087 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.218068    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.218246 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.218222    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/118587b4-0b24-4a51-b462-68bce9602a0a-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.232299 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.232263    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn26p\" (UniqueName: \"kubernetes.io/projected/118587b4-0b24-4a51-b462-68bce9602a0a-kube-api-access-jn26p\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.232970 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.232952    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/118587b4-0b24-4a51-b462-68bce9602a0a-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.281229 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.281197    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:10.387759 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.387517    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"]
Oct 30 19:51:10.399261 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.399229    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.399647 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.399617    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"]
Oct 30 19:51:10.409549 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.409526    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a1d885-openshift-default-dockercfg-stn6c\""
Oct 30 19:51:10.520531 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.520458    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.520531 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.520508    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx29s\" (UniqueName: \"kubernetes.io/projected/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-kube-api-access-bx29s\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.520733 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.520615    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.520733 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.520654    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.520814 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.520744    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.520814 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.520781    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.520814 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.520808    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.520946 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.520846    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.521011 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.520994    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.599735 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.599705    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"]
Oct 30 19:51:10.601513 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:51:10.601480    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118587b4_0b24_4a51_b462_68bce9602a0a.slice/crio-4f89c7df5d895b4bae76089d758980690f2d639a33b8918d0d3f19ff6dd28d68 WatchSource:0}: Error finding container 4f89c7df5d895b4bae76089d758980690f2d639a33b8918d0d3f19ff6dd28d68: Status 404 returned error can't find the container with id 4f89c7df5d895b4bae76089d758980690f2d639a33b8918d0d3f19ff6dd28d68
Oct 30 19:51:10.604042 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.604014    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:51:10.604101 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.604076    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:51:10.604142 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.604105    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:51:10.622020 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.621991    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.622117 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.622041    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.622117 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.622067    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.622117 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.622102    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.622282 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.622207    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.622282 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.622267    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx29s\" (UniqueName: \"kubernetes.io/projected/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-kube-api-access-bx29s\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.622379 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.622317    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.622379 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.622357    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.622485 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.622407    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.622547 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.622500    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.622604 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.622555    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.622727 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.622689    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.622820 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.622800    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.623142 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.623114    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.624728 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.624678    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.625068 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.625047    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.637902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.637875    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.638096 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.638078    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx29s\" (UniqueName: \"kubernetes.io/projected/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-kube-api-access-bx29s\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:10.720042 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:10.720005    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:11.021174 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:11.021140    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"]
Oct 30 19:51:11.021519 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:51:11.021174    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2d0b1a_f650_46ad_bb7f_325ba702a89d.slice/crio-419630ef5148acc98ca587b4bb0f832dc414abc435adeb64388716ccb2b14222 WatchSource:0}: Error finding container 419630ef5148acc98ca587b4bb0f832dc414abc435adeb64388716ccb2b14222: Status 404 returned error can't find the container with id 419630ef5148acc98ca587b4bb0f832dc414abc435adeb64388716ccb2b14222
Oct 30 19:51:11.023945 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:11.023910    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:51:11.024036 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:11.023997    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:51:11.024078 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:11.024042    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:51:11.327252 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:11.327168    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb" event={"ID":"3c2d0b1a-f650-46ad-bb7f-325ba702a89d","Type":"ContainerStarted","Data":"21c1795e95e7b4253754d96de06c7ba7d51313fa7e898f2c75526f7a859ce3f1"}
Oct 30 19:51:11.327252 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:11.327226    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb" event={"ID":"3c2d0b1a-f650-46ad-bb7f-325ba702a89d","Type":"ContainerStarted","Data":"419630ef5148acc98ca587b4bb0f832dc414abc435adeb64388716ccb2b14222"}
Oct 30 19:51:11.329194 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:11.329161    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z" event={"ID":"118587b4-0b24-4a51-b462-68bce9602a0a","Type":"ContainerStarted","Data":"4d2f5bb04ebd505550e9415566f5fca3e4525d08207da928065ccb63134a9bad"}
Oct 30 19:51:11.329336 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:11.329217    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z" event={"ID":"118587b4-0b24-4a51-b462-68bce9602a0a","Type":"ContainerStarted","Data":"4f89c7df5d895b4bae76089d758980690f2d639a33b8918d0d3f19ff6dd28d68"}
Oct 30 19:51:11.355890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:11.352818    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb" podStartSLOduration=1.352801073 podStartE2EDuration="1.352801073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:11.349244922 +0000 UTC m=+1271.241859259" watchObservedRunningTime="2026-10-30 19:51:11.352801073 +0000 UTC m=+1271.245415400"
Oct 30 19:51:11.385166 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:11.385103    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z" podStartSLOduration=2.38508493 podStartE2EDuration="2.38508493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:11.37662846 +0000 UTC m=+1271.269242800" watchObservedRunningTime="2026-10-30 19:51:11.38508493 +0000 UTC m=+1271.277699253"
Oct 30 19:51:11.721143 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:11.721103    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:12.299365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:12.299036    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:51:12.299365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:12.299096    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z" podUID="118587b4-0b24-4a51-b462-68bce9602a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 19:51:12.305974 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:12.305940    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:12.722772 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:12.722734    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:51:12.722962 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:12.722786    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb" podUID="3c2d0b1a-f650-46ad-bb7f-325ba702a89d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:51:13.293247 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:13.291267    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:51:13.293247 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:13.291332    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z" podUID="118587b4-0b24-4a51-b462-68bce9602a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 19:51:13.721293 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:13.721251    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:51:13.721688 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:13.721320    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb" podUID="3c2d0b1a-f650-46ad-bb7f-325ba702a89d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:51:14.301916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:14.284981    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:51:14.301916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:14.285049    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z" podUID="118587b4-0b24-4a51-b462-68bce9602a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 19:51:14.720501 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:14.720459    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:51:14.720682 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:14.720514    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb" podUID="3c2d0b1a-f650-46ad-bb7f-325ba702a89d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:51:15.286886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:15.286364    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:51:15.286886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:15.286424    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z" podUID="118587b4-0b24-4a51-b462-68bce9602a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 19:51:15.720885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:15.720590    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:51:15.720885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:15.720665    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb" podUID="3c2d0b1a-f650-46ad-bb7f-325ba702a89d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:51:16.302846 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:16.302381    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:51:16.302846 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:16.302443    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z" podUID="118587b4-0b24-4a51-b462-68bce9602a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 19:51:16.721095 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:16.721063    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:51:16.721246 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:16.721115    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb" podUID="3c2d0b1a-f650-46ad-bb7f-325ba702a89d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:51:17.282199 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:17.282151    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:51:17.282413 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:17.282219    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z" podUID="118587b4-0b24-4a51-b462-68bce9602a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 19:51:17.721271 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:17.721230    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:51:17.721744 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:17.721301    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb" podUID="3c2d0b1a-f650-46ad-bb7f-325ba702a89d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:51:18.285604 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:18.285569    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:18.286051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:18.286030    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:18.286729 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:18.286712    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 19:51:18.725343 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:18.725314    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:18.725754 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:18.725579    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:18.726219 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:18.726200    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 19:51:25.791119 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:25.791089    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"]
Oct 30 19:51:25.797584 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:25.797551    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:25.812770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:25.812747    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5f933e-openshift-default-dockercfg-dspvq\""
Oct 30 19:51:25.874277 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:25.874247    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"]
Oct 30 19:51:25.967940 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:25.967908    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:25.968104 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:25.967950    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:25.968104 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:25.968071    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:25.968104 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:25.968098    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:25.968228 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:25.968123    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:25.968263 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:25.968233    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:25.968350 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:25.968267    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:25.968350 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:25.968289    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:25.968424 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:25.968392    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8nrz\" (UniqueName: \"kubernetes.io/projected/a247c39a-03e3-4f42-a939-3c80ecc3bff5-kube-api-access-r8nrz\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.069846 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.069743    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.069846 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.069814    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.070093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.069853    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.070093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.069924    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.070093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.069961    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.070093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.069994    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.070093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.070064    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8nrz\" (UniqueName: \"kubernetes.io/projected/a247c39a-03e3-4f42-a939-3c80ecc3bff5-kube-api-access-r8nrz\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.070292 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.070178    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.070292 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.070222    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.070540 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.070514    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.070649 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.070558    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.070649 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.070593    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.070763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.070740    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.070900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.070875    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.072593 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.072570    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.072689 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.072671    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.085531 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.085510    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.085697 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.085669    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8nrz\" (UniqueName: \"kubernetes.io/projected/a247c39a-03e3-4f42-a939-3c80ecc3bff5-kube-api-access-r8nrz\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.114347 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.114318    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:26.408611 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.408572    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"]
Oct 30 19:51:26.412296 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:51:26.412268    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda247c39a_03e3_4f42_a939_3c80ecc3bff5.slice/crio-21c60ccc9aa26bd0081ca2f2a0136671d719898670c5c2fc405c8091139bca18 WatchSource:0}: Error finding container 21c60ccc9aa26bd0081ca2f2a0136671d719898670c5c2fc405c8091139bca18: Status 404 returned error can't find the container with id 21c60ccc9aa26bd0081ca2f2a0136671d719898670c5c2fc405c8091139bca18
Oct 30 19:51:26.415527 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.415497    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:51:26.415614 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.415576    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:51:26.415683 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.415621    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:51:26.480873 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:26.480831    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt" event={"ID":"a247c39a-03e3-4f42-a939-3c80ecc3bff5","Type":"ContainerStarted","Data":"21c60ccc9aa26bd0081ca2f2a0136671d719898670c5c2fc405c8091139bca18"}
Oct 30 19:51:27.489219 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:27.489176    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt" event={"ID":"a247c39a-03e3-4f42-a939-3c80ecc3bff5","Type":"ContainerStarted","Data":"5b30dde630cc917ab7e0fb595d4359e80163a118eb824b7a33d5cfb1994fc87f"}
Oct 30 19:51:27.516266 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:27.516216    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt" podStartSLOduration=2.5162004639999997 podStartE2EDuration="2.516200464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:27.513297122 +0000 UTC m=+1287.405911447" watchObservedRunningTime="2026-10-30 19:51:27.516200464 +0000 UTC m=+1287.408814788"
Oct 30 19:51:28.115926 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.115888    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:28.117385 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.117359    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:51:28.117513 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.117422    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt" podUID="a247c39a-03e3-4f42-a939-3c80ecc3bff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 19:51:28.507277 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.507198    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"]
Oct 30 19:51:28.511508 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.511487    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"
Oct 30 19:51:28.517636 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.517609    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:51:28.517894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.517668    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:51:28.518011 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.517667    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:51:28.518068 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.518041    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:51:28.521387 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.521364    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"]
Oct 30 19:51:28.601356 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.601319    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86625a00-2726-4009-a7a0-456e07604687-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e5e7a13-2wdv2\" (UID: \"86625a00-2726-4009-a7a0-456e07604687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"
Oct 30 19:51:28.601356 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.601362    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86625a00-2726-4009-a7a0-456e07604687-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e5e7a13-2wdv2\" (UID: \"86625a00-2726-4009-a7a0-456e07604687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"
Oct 30 19:51:28.601662 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.601639    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kwfx\" (UniqueName: \"kubernetes.io/projected/86625a00-2726-4009-a7a0-456e07604687-kube-api-access-8kwfx\") pod \"maas-api-revoke-sub-7e5e7a13-2wdv2\" (UID: \"86625a00-2726-4009-a7a0-456e07604687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"
Oct 30 19:51:28.702568 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.702530    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86625a00-2726-4009-a7a0-456e07604687-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e5e7a13-2wdv2\" (UID: \"86625a00-2726-4009-a7a0-456e07604687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"
Oct 30 19:51:28.702568 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.702566    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86625a00-2726-4009-a7a0-456e07604687-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e5e7a13-2wdv2\" (UID: \"86625a00-2726-4009-a7a0-456e07604687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"
Oct 30 19:51:28.702784 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.702717    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kwfx\" (UniqueName: \"kubernetes.io/projected/86625a00-2726-4009-a7a0-456e07604687-kube-api-access-8kwfx\") pod \"maas-api-revoke-sub-7e5e7a13-2wdv2\" (UID: \"86625a00-2726-4009-a7a0-456e07604687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"
Oct 30 19:51:28.703350 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.703326    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86625a00-2726-4009-a7a0-456e07604687-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e5e7a13-2wdv2\" (UID: \"86625a00-2726-4009-a7a0-456e07604687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"
Oct 30 19:51:28.705116 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.705097    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86625a00-2726-4009-a7a0-456e07604687-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e5e7a13-2wdv2\" (UID: \"86625a00-2726-4009-a7a0-456e07604687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"
Oct 30 19:51:28.715103 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.715078    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kwfx\" (UniqueName: \"kubernetes.io/projected/86625a00-2726-4009-a7a0-456e07604687-kube-api-access-8kwfx\") pod \"maas-api-revoke-sub-7e5e7a13-2wdv2\" (UID: \"86625a00-2726-4009-a7a0-456e07604687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"
Oct 30 19:51:28.827216 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:28.827132    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"
Oct 30 19:51:29.114796 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:29.114760    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:51:29.114977 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:29.114839    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt" podUID="a247c39a-03e3-4f42-a939-3c80ecc3bff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 19:51:29.143043 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:29.143010    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"]
Oct 30 19:51:29.144067 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:51:29.144033    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86625a00_2726_4009_a7a0_456e07604687.slice/crio-97031cc7ebb380f17a31d88b5e584db601d0f541016ed8bcdc832d629bcc30a6 WatchSource:0}: Error finding container 97031cc7ebb380f17a31d88b5e584db601d0f541016ed8bcdc832d629bcc30a6: Status 404 returned error can't find the container with id 97031cc7ebb380f17a31d88b5e584db601d0f541016ed8bcdc832d629bcc30a6
Oct 30 19:51:29.146043 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:29.146024    4823 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:51:29.509461 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:29.509364    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2" event={"ID":"86625a00-2726-4009-a7a0-456e07604687","Type":"ContainerStarted","Data":"97031cc7ebb380f17a31d88b5e584db601d0f541016ed8bcdc832d629bcc30a6"}
Oct 30 19:51:30.115023 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:30.114993    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:51:30.115198 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:30.115045    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt" podUID="a247c39a-03e3-4f42-a939-3c80ecc3bff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 19:51:31.114984 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:31.114952    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:51:31.115369 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:31.115003    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt" podUID="a247c39a-03e3-4f42-a939-3c80ecc3bff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 19:51:32.014369 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:51:32.014332    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:51:32.114944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:32.114912    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:51:32.115115 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:32.114963    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt" podUID="a247c39a-03e3-4f42-a939-3c80ecc3bff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 19:51:33.119368 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:33.119338    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:33.119755 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:33.119667    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:33.120902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:33.120882    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 19:51:34.598940 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:34.598904    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:51:36.413727 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:36.413645    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"]
Oct 30 19:51:36.417941 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:36.417906    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"
Oct 30 19:51:36.437473 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:36.437445    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"]
Oct 30 19:51:36.523166 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:36.523106    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5da1768-d7b5-4667-bea4-8136dd52fe63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a947d1a3-vvrdx\" (UID: \"e5da1768-d7b5-4667-bea4-8136dd52fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"
Oct 30 19:51:36.523362 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:36.523273    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b9dq\" (UniqueName: \"kubernetes.io/projected/e5da1768-d7b5-4667-bea4-8136dd52fe63-kube-api-access-7b9dq\") pod \"maas-api-revoke-sub-a947d1a3-vvrdx\" (UID: \"e5da1768-d7b5-4667-bea4-8136dd52fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"
Oct 30 19:51:36.523362 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:36.523321    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5da1768-d7b5-4667-bea4-8136dd52fe63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a947d1a3-vvrdx\" (UID: \"e5da1768-d7b5-4667-bea4-8136dd52fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"
Oct 30 19:51:36.555266 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:36.555233    4823 generic.go:356] "Generic (PLEG): container finished" podID="86625a00-2726-4009-a7a0-456e07604687" containerID="53362280dd5a401fc0ce93aad0a804283e214a038d65cff9187e93ebe15a1417" exitCode=0
Oct 30 19:51:36.555434 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:36.555313    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2" event={"ID":"86625a00-2726-4009-a7a0-456e07604687","Type":"ContainerDied","Data":"53362280dd5a401fc0ce93aad0a804283e214a038d65cff9187e93ebe15a1417"}
Oct 30 19:51:36.624741 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:36.624708    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5da1768-d7b5-4667-bea4-8136dd52fe63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a947d1a3-vvrdx\" (UID: \"e5da1768-d7b5-4667-bea4-8136dd52fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"
Oct 30 19:51:36.624942 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:36.624807    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b9dq\" (UniqueName: \"kubernetes.io/projected/e5da1768-d7b5-4667-bea4-8136dd52fe63-kube-api-access-7b9dq\") pod \"maas-api-revoke-sub-a947d1a3-vvrdx\" (UID: \"e5da1768-d7b5-4667-bea4-8136dd52fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"
Oct 30 19:51:36.624942 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:36.624836    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5da1768-d7b5-4667-bea4-8136dd52fe63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a947d1a3-vvrdx\" (UID: \"e5da1768-d7b5-4667-bea4-8136dd52fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"
Oct 30 19:51:36.625558 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:36.625537    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5da1768-d7b5-4667-bea4-8136dd52fe63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a947d1a3-vvrdx\" (UID: \"e5da1768-d7b5-4667-bea4-8136dd52fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"
Oct 30 19:51:36.627705 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:36.627680    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5da1768-d7b5-4667-bea4-8136dd52fe63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a947d1a3-vvrdx\" (UID: \"e5da1768-d7b5-4667-bea4-8136dd52fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"
Oct 30 19:51:36.681603 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:36.681531    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b9dq\" (UniqueName: \"kubernetes.io/projected/e5da1768-d7b5-4667-bea4-8136dd52fe63-kube-api-access-7b9dq\") pod \"maas-api-revoke-sub-a947d1a3-vvrdx\" (UID: \"e5da1768-d7b5-4667-bea4-8136dd52fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"
Oct 30 19:51:36.731194 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:36.731161    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"
Oct 30 19:51:37.029626 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:37.029597    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"]
Oct 30 19:51:37.032111 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:51:37.032085    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5da1768_d7b5_4667_bea4_8136dd52fe63.slice/crio-1ad45079d50db82d7d21001b176065fe567029cfc5ae81467c45d00733254307 WatchSource:0}: Error finding container 1ad45079d50db82d7d21001b176065fe567029cfc5ae81467c45d00733254307: Status 404 returned error can't find the container with id 1ad45079d50db82d7d21001b176065fe567029cfc5ae81467c45d00733254307
Oct 30 19:51:37.564411 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:37.564372    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx" event={"ID":"e5da1768-d7b5-4667-bea4-8136dd52fe63","Type":"ContainerStarted","Data":"742c89f9b096c3bd610fc3ad60bf0ae4ea635715444b3d42a0edff05adba3230"}
Oct 30 19:51:37.564878 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:37.564424    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx" event={"ID":"e5da1768-d7b5-4667-bea4-8136dd52fe63","Type":"ContainerStarted","Data":"1ad45079d50db82d7d21001b176065fe567029cfc5ae81467c45d00733254307"}
Oct 30 19:51:37.607626 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:37.607561    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx" podStartSLOduration=1.607543156 podStartE2EDuration="1.607543156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:37.603712103 +0000 UTC m=+1297.496326449" watchObservedRunningTime="2026-10-30 19:51:37.607543156 +0000 UTC m=+1297.500157481"
Oct 30 19:51:37.721223 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:37.721198    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"
Oct 30 19:51:37.839042 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:37.839013    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86625a00-2726-4009-a7a0-456e07604687-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86625a00-2726-4009-a7a0-456e07604687-maas-api-openshift-ca\") pod \"86625a00-2726-4009-a7a0-456e07604687\" (UID: \"86625a00-2726-4009-a7a0-456e07604687\") "
Oct 30 19:51:37.839208 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:37.839098    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86625a00-2726-4009-a7a0-456e07604687-kube-api-access-8kwfx\" (UniqueName: \"kubernetes.io/projected/86625a00-2726-4009-a7a0-456e07604687-kube-api-access-8kwfx\") pod \"86625a00-2726-4009-a7a0-456e07604687\" (UID: \"86625a00-2726-4009-a7a0-456e07604687\") "
Oct 30 19:51:37.839267 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:37.839244    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86625a00-2726-4009-a7a0-456e07604687-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86625a00-2726-4009-a7a0-456e07604687-maas-api-xks-ca\") pod \"86625a00-2726-4009-a7a0-456e07604687\" (UID: \"86625a00-2726-4009-a7a0-456e07604687\") "
Oct 30 19:51:37.839422 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:37.839395    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86625a00-2726-4009-a7a0-456e07604687-maas-api-openshift-ca" pod "86625a00-2726-4009-a7a0-456e07604687" (UID: "86625a00-2726-4009-a7a0-456e07604687"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:37.839745 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:37.839716    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86625a00-2726-4009-a7a0-456e07604687-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:37.841272 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:37.841244    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86625a00-2726-4009-a7a0-456e07604687-maas-api-xks-ca" pod "86625a00-2726-4009-a7a0-456e07604687" (UID: "86625a00-2726-4009-a7a0-456e07604687"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:37.841369 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:37.841312    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86625a00-2726-4009-a7a0-456e07604687-kube-api-access-8kwfx" pod "86625a00-2726-4009-a7a0-456e07604687" (UID: "86625a00-2726-4009-a7a0-456e07604687"). InnerVolumeSpecName "kube-api-access-8kwfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:37.940665 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:37.940586    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kwfx\" (UniqueName: \"kubernetes.io/projected/86625a00-2726-4009-a7a0-456e07604687-kube-api-access-8kwfx\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:37.940665 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:37.940619    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86625a00-2726-4009-a7a0-456e07604687-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:38.572832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:38.572751    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"
Oct 30 19:51:38.572832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:38.572765    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2" event={"ID":"86625a00-2726-4009-a7a0-456e07604687","Type":"ContainerDied","Data":"97031cc7ebb380f17a31d88b5e584db601d0f541016ed8bcdc832d629bcc30a6"}
Oct 30 19:51:38.572832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:38.572820    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97031cc7ebb380f17a31d88b5e584db601d0f541016ed8bcdc832d629bcc30a6"
Oct 30 19:51:38.574665 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:38.574633    4823 generic.go:356] "Generic (PLEG): container finished" podID="e5da1768-d7b5-4667-bea4-8136dd52fe63" containerID="742c89f9b096c3bd610fc3ad60bf0ae4ea635715444b3d42a0edff05adba3230" exitCode=0
Oct 30 19:51:38.574805 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:38.574738    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx" event={"ID":"e5da1768-d7b5-4667-bea4-8136dd52fe63","Type":"ContainerDied","Data":"742c89f9b096c3bd610fc3ad60bf0ae4ea635715444b3d42a0edff05adba3230"}
Oct 30 19:51:39.776680 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:39.776654    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"
Oct 30 19:51:39.866112 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:39.866084    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5da1768-d7b5-4667-bea4-8136dd52fe63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5da1768-d7b5-4667-bea4-8136dd52fe63-maas-api-openshift-ca\") pod \"e5da1768-d7b5-4667-bea4-8136dd52fe63\" (UID: \"e5da1768-d7b5-4667-bea4-8136dd52fe63\") "
Oct 30 19:51:39.866315 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:39.866158    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5da1768-d7b5-4667-bea4-8136dd52fe63-kube-api-access-7b9dq\" (UniqueName: \"kubernetes.io/projected/e5da1768-d7b5-4667-bea4-8136dd52fe63-kube-api-access-7b9dq\") pod \"e5da1768-d7b5-4667-bea4-8136dd52fe63\" (UID: \"e5da1768-d7b5-4667-bea4-8136dd52fe63\") "
Oct 30 19:51:39.866315 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:39.866244    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5da1768-d7b5-4667-bea4-8136dd52fe63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5da1768-d7b5-4667-bea4-8136dd52fe63-maas-api-xks-ca\") pod \"e5da1768-d7b5-4667-bea4-8136dd52fe63\" (UID: \"e5da1768-d7b5-4667-bea4-8136dd52fe63\") "
Oct 30 19:51:39.866581 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:39.866554    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5da1768-d7b5-4667-bea4-8136dd52fe63-maas-api-openshift-ca" pod "e5da1768-d7b5-4667-bea4-8136dd52fe63" (UID: "e5da1768-d7b5-4667-bea4-8136dd52fe63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:39.868296 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:39.868275    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5da1768-d7b5-4667-bea4-8136dd52fe63-maas-api-xks-ca" pod "e5da1768-d7b5-4667-bea4-8136dd52fe63" (UID: "e5da1768-d7b5-4667-bea4-8136dd52fe63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:39.868520 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:39.868502    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5da1768-d7b5-4667-bea4-8136dd52fe63-kube-api-access-7b9dq" pod "e5da1768-d7b5-4667-bea4-8136dd52fe63" (UID: "e5da1768-d7b5-4667-bea4-8136dd52fe63"). InnerVolumeSpecName "kube-api-access-7b9dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:39.967637 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:39.967606    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5da1768-d7b5-4667-bea4-8136dd52fe63-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:39.967637 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:39.967634    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b9dq\" (UniqueName: \"kubernetes.io/projected/e5da1768-d7b5-4667-bea4-8136dd52fe63-kube-api-access-7b9dq\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:39.967903 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:39.967650    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5da1768-d7b5-4667-bea4-8136dd52fe63-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:40.589104 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:40.589021    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx" event={"ID":"e5da1768-d7b5-4667-bea4-8136dd52fe63","Type":"ContainerDied","Data":"1ad45079d50db82d7d21001b176065fe567029cfc5ae81467c45d00733254307"}
Oct 30 19:51:40.589104 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:40.589048    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"
Oct 30 19:51:40.589104 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:40.589068    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ad45079d50db82d7d21001b176065fe567029cfc5ae81467c45d00733254307"
Oct 30 19:51:43.656004 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.655971    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"]
Oct 30 19:51:43.656553 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.656538    4823 cpu_manager.go:414] "containerMap: removing container" podUID="e5da1768-d7b5-4667-bea4-8136dd52fe63" containerName="revoke-keys"
Oct 30 19:51:43.656597 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.656555    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5da1768-d7b5-4667-bea4-8136dd52fe63" containerName="revoke-keys"
Oct 30 19:51:43.656597 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.656573    4823 cpu_manager.go:414] "containerMap: removing container" podUID="86625a00-2726-4009-a7a0-456e07604687" containerName="revoke-keys"
Oct 30 19:51:43.656597 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.656579    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86625a00-2726-4009-a7a0-456e07604687" containerName="revoke-keys"
Oct 30 19:51:43.656685 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.656677    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="86625a00-2726-4009-a7a0-456e07604687" containerName="revoke-keys"
Oct 30 19:51:43.656722 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.656692    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5da1768-d7b5-4667-bea4-8136dd52fe63" containerName="revoke-keys"
Oct 30 19:51:43.660341 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.660327    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"
Oct 30 19:51:43.663903 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.663875    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:51:43.664184 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.664167    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:51:43.671071 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.671040    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"]
Oct 30 19:51:43.764348 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.764315    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:51:43.764501 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.764352    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:51:43.812837 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.812807    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4c0f22-4317-4c34-ada6-2f40f7140a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2877b35f-7vcwg\" (UID: \"dc4c0f22-4317-4c34-ada6-2f40f7140a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"
Oct 30 19:51:43.813025 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.812855    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4c0f22-4317-4c34-ada6-2f40f7140a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2877b35f-7vcwg\" (UID: \"dc4c0f22-4317-4c34-ada6-2f40f7140a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"
Oct 30 19:51:43.813025 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.813001    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r94dw\" (UniqueName: \"kubernetes.io/projected/dc4c0f22-4317-4c34-ada6-2f40f7140a10-kube-api-access-r94dw\") pod \"maas-api-revoke-sub-2877b35f-7vcwg\" (UID: \"dc4c0f22-4317-4c34-ada6-2f40f7140a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"
Oct 30 19:51:43.914568 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.914491    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4c0f22-4317-4c34-ada6-2f40f7140a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2877b35f-7vcwg\" (UID: \"dc4c0f22-4317-4c34-ada6-2f40f7140a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"
Oct 30 19:51:43.914568 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.914547    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4c0f22-4317-4c34-ada6-2f40f7140a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2877b35f-7vcwg\" (UID: \"dc4c0f22-4317-4c34-ada6-2f40f7140a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"
Oct 30 19:51:43.914789 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.914610    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r94dw\" (UniqueName: \"kubernetes.io/projected/dc4c0f22-4317-4c34-ada6-2f40f7140a10-kube-api-access-r94dw\") pod \"maas-api-revoke-sub-2877b35f-7vcwg\" (UID: \"dc4c0f22-4317-4c34-ada6-2f40f7140a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"
Oct 30 19:51:43.915264 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.915237    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4c0f22-4317-4c34-ada6-2f40f7140a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2877b35f-7vcwg\" (UID: \"dc4c0f22-4317-4c34-ada6-2f40f7140a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"
Oct 30 19:51:43.916874 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.916841    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4c0f22-4317-4c34-ada6-2f40f7140a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2877b35f-7vcwg\" (UID: \"dc4c0f22-4317-4c34-ada6-2f40f7140a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"
Oct 30 19:51:43.936787 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.936766    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r94dw\" (UniqueName: \"kubernetes.io/projected/dc4c0f22-4317-4c34-ada6-2f40f7140a10-kube-api-access-r94dw\") pod \"maas-api-revoke-sub-2877b35f-7vcwg\" (UID: \"dc4c0f22-4317-4c34-ada6-2f40f7140a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"
Oct 30 19:51:43.975479 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:43.975444    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"
Oct 30 19:51:44.264316 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:44.264290    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"]
Oct 30 19:51:44.266446 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:51:44.266419    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4c0f22_4317_4c34_ada6_2f40f7140a10.slice/crio-7412432dd36b207654f63607d23bd1256e44a3681e0df63b7d97fc7d64b4d76f WatchSource:0}: Error finding container 7412432dd36b207654f63607d23bd1256e44a3681e0df63b7d97fc7d64b4d76f: Status 404 returned error can't find the container with id 7412432dd36b207654f63607d23bd1256e44a3681e0df63b7d97fc7d64b4d76f
Oct 30 19:51:44.618843 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:44.618809    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg" event={"ID":"dc4c0f22-4317-4c34-ada6-2f40f7140a10","Type":"ContainerStarted","Data":"1fac3582036d1604b9824547cd3e41a7e20e0bdf49914824a31a8c99d3302b1c"}
Oct 30 19:51:44.619044 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:44.618886    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg" event={"ID":"dc4c0f22-4317-4c34-ada6-2f40f7140a10","Type":"ContainerStarted","Data":"7412432dd36b207654f63607d23bd1256e44a3681e0df63b7d97fc7d64b4d76f"}
Oct 30 19:51:44.664930 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:44.664875    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg" podStartSLOduration=1.664844177 podStartE2EDuration="1.664844177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:44.662165251 +0000 UTC m=+1304.554779576" watchObservedRunningTime="2026-10-30 19:51:44.664844177 +0000 UTC m=+1304.557458502"
Oct 30 19:51:45.012743 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:45.011753    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"]
Oct 30 19:51:45.017139 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:45.017110    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"
Oct 30 19:51:45.035596 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:45.035566    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"]
Oct 30 19:51:45.131037 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:45.131005    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f4e928-2f39-4273-b567-e21b841e7ce5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-450f5839-7vkhj\" (UID: \"e9f4e928-2f39-4273-b567-e21b841e7ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"
Oct 30 19:51:45.131203 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:45.131055    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4gjv\" (UniqueName: \"kubernetes.io/projected/e9f4e928-2f39-4273-b567-e21b841e7ce5-kube-api-access-z4gjv\") pod \"maas-api-revoke-sub-450f5839-7vkhj\" (UID: \"e9f4e928-2f39-4273-b567-e21b841e7ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"
Oct 30 19:51:45.131203 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:45.131200    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f4e928-2f39-4273-b567-e21b841e7ce5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-450f5839-7vkhj\" (UID: \"e9f4e928-2f39-4273-b567-e21b841e7ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"
Oct 30 19:51:45.238732 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:45.238696    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f4e928-2f39-4273-b567-e21b841e7ce5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-450f5839-7vkhj\" (UID: \"e9f4e928-2f39-4273-b567-e21b841e7ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"
Oct 30 19:51:45.239005 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:45.238984    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f4e928-2f39-4273-b567-e21b841e7ce5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-450f5839-7vkhj\" (UID: \"e9f4e928-2f39-4273-b567-e21b841e7ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"
Oct 30 19:51:45.239073 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:45.239040    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4gjv\" (UniqueName: \"kubernetes.io/projected/e9f4e928-2f39-4273-b567-e21b841e7ce5-kube-api-access-z4gjv\") pod \"maas-api-revoke-sub-450f5839-7vkhj\" (UID: \"e9f4e928-2f39-4273-b567-e21b841e7ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"
Oct 30 19:51:45.239668 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:45.239639    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f4e928-2f39-4273-b567-e21b841e7ce5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-450f5839-7vkhj\" (UID: \"e9f4e928-2f39-4273-b567-e21b841e7ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"
Oct 30 19:51:45.241340 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:45.241316    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f4e928-2f39-4273-b567-e21b841e7ce5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-450f5839-7vkhj\" (UID: \"e9f4e928-2f39-4273-b567-e21b841e7ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"
Oct 30 19:51:45.251083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:45.251060    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4gjv\" (UniqueName: \"kubernetes.io/projected/e9f4e928-2f39-4273-b567-e21b841e7ce5-kube-api-access-z4gjv\") pod \"maas-api-revoke-sub-450f5839-7vkhj\" (UID: \"e9f4e928-2f39-4273-b567-e21b841e7ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"
Oct 30 19:51:45.332401 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:45.332308    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"
Oct 30 19:51:45.628714 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:45.628681    4823 generic.go:356] "Generic (PLEG): container finished" podID="dc4c0f22-4317-4c34-ada6-2f40f7140a10" containerID="1fac3582036d1604b9824547cd3e41a7e20e0bdf49914824a31a8c99d3302b1c" exitCode=0
Oct 30 19:51:45.628852 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:45.628761    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg" event={"ID":"dc4c0f22-4317-4c34-ada6-2f40f7140a10","Type":"ContainerDied","Data":"1fac3582036d1604b9824547cd3e41a7e20e0bdf49914824a31a8c99d3302b1c"}
Oct 30 19:51:45.629217 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:45.629195    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"]
Oct 30 19:51:45.631147 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:51:45.631122    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f4e928_2f39_4273_b567_e21b841e7ce5.slice/crio-ce59625b4eec5ed3454135b8be18af5edc28220193e2806b8fb58d03d4edaba5 WatchSource:0}: Error finding container ce59625b4eec5ed3454135b8be18af5edc28220193e2806b8fb58d03d4edaba5: Status 404 returned error can't find the container with id ce59625b4eec5ed3454135b8be18af5edc28220193e2806b8fb58d03d4edaba5
Oct 30 19:51:46.636590 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:46.636556    4823 generic.go:356] "Generic (PLEG): container finished" podID="e9f4e928-2f39-4273-b567-e21b841e7ce5" containerID="27f79e82c465856bafd07f5418bfc4720335b79fa9dc0a862077462991c17d99" exitCode=0
Oct 30 19:51:46.637000 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:46.636634    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj" event={"ID":"e9f4e928-2f39-4273-b567-e21b841e7ce5","Type":"ContainerDied","Data":"27f79e82c465856bafd07f5418bfc4720335b79fa9dc0a862077462991c17d99"}
Oct 30 19:51:46.637000 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:46.636686    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj" event={"ID":"e9f4e928-2f39-4273-b567-e21b841e7ce5","Type":"ContainerStarted","Data":"ce59625b4eec5ed3454135b8be18af5edc28220193e2806b8fb58d03d4edaba5"}
Oct 30 19:51:46.795738 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:46.795709    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"
Oct 30 19:51:46.859242 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:46.859213    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc4c0f22-4317-4c34-ada6-2f40f7140a10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4c0f22-4317-4c34-ada6-2f40f7140a10-maas-api-openshift-ca\") pod \"dc4c0f22-4317-4c34-ada6-2f40f7140a10\" (UID: \"dc4c0f22-4317-4c34-ada6-2f40f7140a10\") "
Oct 30 19:51:46.859400 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:46.859310    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4c0f22-4317-4c34-ada6-2f40f7140a10-kube-api-access-r94dw\" (UniqueName: \"kubernetes.io/projected/dc4c0f22-4317-4c34-ada6-2f40f7140a10-kube-api-access-r94dw\") pod \"dc4c0f22-4317-4c34-ada6-2f40f7140a10\" (UID: \"dc4c0f22-4317-4c34-ada6-2f40f7140a10\") "
Oct 30 19:51:46.859400 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:46.859382    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc4c0f22-4317-4c34-ada6-2f40f7140a10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4c0f22-4317-4c34-ada6-2f40f7140a10-maas-api-xks-ca\") pod \"dc4c0f22-4317-4c34-ada6-2f40f7140a10\" (UID: \"dc4c0f22-4317-4c34-ada6-2f40f7140a10\") "
Oct 30 19:51:46.859558 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:46.859536    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4c0f22-4317-4c34-ada6-2f40f7140a10-maas-api-openshift-ca" pod "dc4c0f22-4317-4c34-ada6-2f40f7140a10" (UID: "dc4c0f22-4317-4c34-ada6-2f40f7140a10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:46.859800 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:46.859777    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4c0f22-4317-4c34-ada6-2f40f7140a10-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:46.861449 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:46.861425    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc4c0f22-4317-4c34-ada6-2f40f7140a10-maas-api-xks-ca" pod "dc4c0f22-4317-4c34-ada6-2f40f7140a10" (UID: "dc4c0f22-4317-4c34-ada6-2f40f7140a10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:46.861519 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:46.861468    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4c0f22-4317-4c34-ada6-2f40f7140a10-kube-api-access-r94dw" pod "dc4c0f22-4317-4c34-ada6-2f40f7140a10" (UID: "dc4c0f22-4317-4c34-ada6-2f40f7140a10"). InnerVolumeSpecName "kube-api-access-r94dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:46.960661 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:46.960626    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r94dw\" (UniqueName: \"kubernetes.io/projected/dc4c0f22-4317-4c34-ada6-2f40f7140a10-kube-api-access-r94dw\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:46.960661 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:46.960661    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4c0f22-4317-4c34-ada6-2f40f7140a10-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:47.644303 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:47.644277    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"
Oct 30 19:51:47.644736 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:47.644314    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg" event={"ID":"dc4c0f22-4317-4c34-ada6-2f40f7140a10","Type":"ContainerDied","Data":"7412432dd36b207654f63607d23bd1256e44a3681e0df63b7d97fc7d64b4d76f"}
Oct 30 19:51:47.644736 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:47.644384    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7412432dd36b207654f63607d23bd1256e44a3681e0df63b7d97fc7d64b4d76f"
Oct 30 19:51:47.807017 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:47.806986    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"
Oct 30 19:51:47.871738 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:47.871706    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f4e928-2f39-4273-b567-e21b841e7ce5-kube-api-access-z4gjv\" (UniqueName: \"kubernetes.io/projected/e9f4e928-2f39-4273-b567-e21b841e7ce5-kube-api-access-z4gjv\") pod \"e9f4e928-2f39-4273-b567-e21b841e7ce5\" (UID: \"e9f4e928-2f39-4273-b567-e21b841e7ce5\") "
Oct 30 19:51:47.871951 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:47.871765    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9f4e928-2f39-4273-b567-e21b841e7ce5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f4e928-2f39-4273-b567-e21b841e7ce5-maas-api-xks-ca\") pod \"e9f4e928-2f39-4273-b567-e21b841e7ce5\" (UID: \"e9f4e928-2f39-4273-b567-e21b841e7ce5\") "
Oct 30 19:51:47.871951 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:47.871802    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f4e928-2f39-4273-b567-e21b841e7ce5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f4e928-2f39-4273-b567-e21b841e7ce5-maas-api-openshift-ca\") pod \"e9f4e928-2f39-4273-b567-e21b841e7ce5\" (UID: \"e9f4e928-2f39-4273-b567-e21b841e7ce5\") "
Oct 30 19:51:47.872264 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:47.872233    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f4e928-2f39-4273-b567-e21b841e7ce5-maas-api-openshift-ca" pod "e9f4e928-2f39-4273-b567-e21b841e7ce5" (UID: "e9f4e928-2f39-4273-b567-e21b841e7ce5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:47.872568 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:47.872521    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f4e928-2f39-4273-b567-e21b841e7ce5-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:47.874165 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:47.874129    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f4e928-2f39-4273-b567-e21b841e7ce5-maas-api-xks-ca" pod "e9f4e928-2f39-4273-b567-e21b841e7ce5" (UID: "e9f4e928-2f39-4273-b567-e21b841e7ce5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:47.874481 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:47.874444    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f4e928-2f39-4273-b567-e21b841e7ce5-kube-api-access-z4gjv" pod "e9f4e928-2f39-4273-b567-e21b841e7ce5" (UID: "e9f4e928-2f39-4273-b567-e21b841e7ce5"). InnerVolumeSpecName "kube-api-access-z4gjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:47.973398 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:47.973336    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4gjv\" (UniqueName: \"kubernetes.io/projected/e9f4e928-2f39-4273-b567-e21b841e7ce5-kube-api-access-z4gjv\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:47.973398 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:47.973362    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f4e928-2f39-4273-b567-e21b841e7ce5-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:48.652045 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:48.652018    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"
Oct 30 19:51:48.652492 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:48.652007    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj" event={"ID":"e9f4e928-2f39-4273-b567-e21b841e7ce5","Type":"ContainerDied","Data":"ce59625b4eec5ed3454135b8be18af5edc28220193e2806b8fb58d03d4edaba5"}
Oct 30 19:51:48.652492 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:48.652164    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce59625b4eec5ed3454135b8be18af5edc28220193e2806b8fb58d03d4edaba5"
Oct 30 19:51:51.912720 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:51.912687    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"]
Oct 30 19:51:51.913317 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:51.913300    4823 cpu_manager.go:414] "containerMap: removing container" podUID="dc4c0f22-4317-4c34-ada6-2f40f7140a10" containerName="revoke-keys"
Oct 30 19:51:51.913363 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:51.913318    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc4c0f22-4317-4c34-ada6-2f40f7140a10" containerName="revoke-keys"
Oct 30 19:51:51.913363 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:51.913340    4823 cpu_manager.go:414] "containerMap: removing container" podUID="e9f4e928-2f39-4273-b567-e21b841e7ce5" containerName="revoke-keys"
Oct 30 19:51:51.913363 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:51.913346    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f4e928-2f39-4273-b567-e21b841e7ce5" containerName="revoke-keys"
Oct 30 19:51:51.913513 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:51.913443    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc4c0f22-4317-4c34-ada6-2f40f7140a10" containerName="revoke-keys"
Oct 30 19:51:51.913513 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:51.913456    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f4e928-2f39-4273-b567-e21b841e7ce5" containerName="revoke-keys"
Oct 30 19:51:51.917142 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:51.917125    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"
Oct 30 19:51:51.923259 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:51.923232    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:51:51.923449 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:51.923425    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:51:51.923523 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:51.923457    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:51:51.923574 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:51.923563    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:51:51.926695 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:51.926673    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"]
Oct 30 19:51:52.047739 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:52.047704    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf4xp\" (UniqueName: \"kubernetes.io/projected/27c7794b-a90d-46ca-9667-1729a711596e-kube-api-access-vf4xp\") pod \"maas-api-revoke-sub-4a2bcef1-n5rs5\" (UID: \"27c7794b-a90d-46ca-9667-1729a711596e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"
Oct 30 19:51:52.047952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:52.047750    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c7794b-a90d-46ca-9667-1729a711596e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a2bcef1-n5rs5\" (UID: \"27c7794b-a90d-46ca-9667-1729a711596e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"
Oct 30 19:51:52.047952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:52.047784    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c7794b-a90d-46ca-9667-1729a711596e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a2bcef1-n5rs5\" (UID: \"27c7794b-a90d-46ca-9667-1729a711596e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"
Oct 30 19:51:52.149403 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:52.149329    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf4xp\" (UniqueName: \"kubernetes.io/projected/27c7794b-a90d-46ca-9667-1729a711596e-kube-api-access-vf4xp\") pod \"maas-api-revoke-sub-4a2bcef1-n5rs5\" (UID: \"27c7794b-a90d-46ca-9667-1729a711596e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"
Oct 30 19:51:52.149403 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:52.149376    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c7794b-a90d-46ca-9667-1729a711596e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a2bcef1-n5rs5\" (UID: \"27c7794b-a90d-46ca-9667-1729a711596e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"
Oct 30 19:51:52.149657 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:52.149419    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c7794b-a90d-46ca-9667-1729a711596e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a2bcef1-n5rs5\" (UID: \"27c7794b-a90d-46ca-9667-1729a711596e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"
Oct 30 19:51:52.150356 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:52.150331    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c7794b-a90d-46ca-9667-1729a711596e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a2bcef1-n5rs5\" (UID: \"27c7794b-a90d-46ca-9667-1729a711596e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"
Oct 30 19:51:52.151885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:52.151850    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c7794b-a90d-46ca-9667-1729a711596e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a2bcef1-n5rs5\" (UID: \"27c7794b-a90d-46ca-9667-1729a711596e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"
Oct 30 19:51:52.161402 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:52.161375    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf4xp\" (UniqueName: \"kubernetes.io/projected/27c7794b-a90d-46ca-9667-1729a711596e-kube-api-access-vf4xp\") pod \"maas-api-revoke-sub-4a2bcef1-n5rs5\" (UID: \"27c7794b-a90d-46ca-9667-1729a711596e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"
Oct 30 19:51:52.232389 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:52.232314    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"
Oct 30 19:51:52.529566 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:52.529538    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"]
Oct 30 19:51:52.532680 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:51:52.532648    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c7794b_a90d_46ca_9667_1729a711596e.slice/crio-bc376d476302ca6bd7daf18dc0a975a03065fce3443e2d189acc2ca51053f40b WatchSource:0}: Error finding container bc376d476302ca6bd7daf18dc0a975a03065fce3443e2d189acc2ca51053f40b: Status 404 returned error can't find the container with id bc376d476302ca6bd7daf18dc0a975a03065fce3443e2d189acc2ca51053f40b
Oct 30 19:51:52.691072 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:52.691030    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5" event={"ID":"27c7794b-a90d-46ca-9667-1729a711596e","Type":"ContainerStarted","Data":"bc376d476302ca6bd7daf18dc0a975a03065fce3443e2d189acc2ca51053f40b"}
Oct 30 19:51:53.078051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.078016    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"]
Oct 30 19:51:53.082289 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.082270    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"
Oct 30 19:51:53.100188 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.100151    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"]
Oct 30 19:51:53.264628 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.264550    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b647870-c146-4318-816a-ba78907e8b10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-335206cd-wrfwz\" (UID: \"5b647870-c146-4318-816a-ba78907e8b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"
Oct 30 19:51:53.264628 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.264616    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b647870-c146-4318-816a-ba78907e8b10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-335206cd-wrfwz\" (UID: \"5b647870-c146-4318-816a-ba78907e8b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"
Oct 30 19:51:53.264804 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.264705    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r42j7\" (UniqueName: \"kubernetes.io/projected/5b647870-c146-4318-816a-ba78907e8b10-kube-api-access-r42j7\") pod \"maas-api-revoke-sub-335206cd-wrfwz\" (UID: \"5b647870-c146-4318-816a-ba78907e8b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"
Oct 30 19:51:53.366188 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.366151    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b647870-c146-4318-816a-ba78907e8b10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-335206cd-wrfwz\" (UID: \"5b647870-c146-4318-816a-ba78907e8b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"
Oct 30 19:51:53.366361 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.366233    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b647870-c146-4318-816a-ba78907e8b10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-335206cd-wrfwz\" (UID: \"5b647870-c146-4318-816a-ba78907e8b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"
Oct 30 19:51:53.366361 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.366268    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r42j7\" (UniqueName: \"kubernetes.io/projected/5b647870-c146-4318-816a-ba78907e8b10-kube-api-access-r42j7\") pod \"maas-api-revoke-sub-335206cd-wrfwz\" (UID: \"5b647870-c146-4318-816a-ba78907e8b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"
Oct 30 19:51:53.366943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.366918    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b647870-c146-4318-816a-ba78907e8b10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-335206cd-wrfwz\" (UID: \"5b647870-c146-4318-816a-ba78907e8b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"
Oct 30 19:51:53.368599 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.368574    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b647870-c146-4318-816a-ba78907e8b10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-335206cd-wrfwz\" (UID: \"5b647870-c146-4318-816a-ba78907e8b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"
Oct 30 19:51:53.382752 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.382727    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r42j7\" (UniqueName: \"kubernetes.io/projected/5b647870-c146-4318-816a-ba78907e8b10-kube-api-access-r42j7\") pod \"maas-api-revoke-sub-335206cd-wrfwz\" (UID: \"5b647870-c146-4318-816a-ba78907e8b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"
Oct 30 19:51:53.395613 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.395589    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"
Oct 30 19:51:53.677876 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.677828    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"]
Oct 30 19:51:53.679342 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:51:53.679317    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b647870_c146_4318_816a_ba78907e8b10.slice/crio-f4fd5524bb90a32b419c288c314c33a29e2ac3c146da2b459c9336d9d5e0c850 WatchSource:0}: Error finding container f4fd5524bb90a32b419c288c314c33a29e2ac3c146da2b459c9336d9d5e0c850: Status 404 returned error can't find the container with id f4fd5524bb90a32b419c288c314c33a29e2ac3c146da2b459c9336d9d5e0c850
Oct 30 19:51:53.700648 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.700618    4823 generic.go:356] "Generic (PLEG): container finished" podID="27c7794b-a90d-46ca-9667-1729a711596e" containerID="8b89c56a018fa879ac72b69e436da8549685d90fc729ff837e5432798352cc51" exitCode=0
Oct 30 19:51:53.700762 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.700740    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5" event={"ID":"27c7794b-a90d-46ca-9667-1729a711596e","Type":"ContainerDied","Data":"8b89c56a018fa879ac72b69e436da8549685d90fc729ff837e5432798352cc51"}
Oct 30 19:51:53.702275 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:53.702250    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz" event={"ID":"5b647870-c146-4318-816a-ba78907e8b10","Type":"ContainerStarted","Data":"f4fd5524bb90a32b419c288c314c33a29e2ac3c146da2b459c9336d9d5e0c850"}
Oct 30 19:51:54.712608 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:54.712577    4823 generic.go:356] "Generic (PLEG): container finished" podID="5b647870-c146-4318-816a-ba78907e8b10" containerID="11267ff2c0418c2985e64e73944274ba0e802a28b9883068950834a1a9cbca37" exitCode=0
Oct 30 19:51:54.713134 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:54.712661    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz" event={"ID":"5b647870-c146-4318-816a-ba78907e8b10","Type":"ContainerDied","Data":"11267ff2c0418c2985e64e73944274ba0e802a28b9883068950834a1a9cbca37"}
Oct 30 19:51:54.886648 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:54.886610    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"
Oct 30 19:51:54.890747 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:54.890724    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c7794b-a90d-46ca-9667-1729a711596e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c7794b-a90d-46ca-9667-1729a711596e-maas-api-xks-ca\") pod \"27c7794b-a90d-46ca-9667-1729a711596e\" (UID: \"27c7794b-a90d-46ca-9667-1729a711596e\") "
Oct 30 19:51:54.890923 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:54.890822    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c7794b-a90d-46ca-9667-1729a711596e-kube-api-access-vf4xp\" (UniqueName: \"kubernetes.io/projected/27c7794b-a90d-46ca-9667-1729a711596e-kube-api-access-vf4xp\") pod \"27c7794b-a90d-46ca-9667-1729a711596e\" (UID: \"27c7794b-a90d-46ca-9667-1729a711596e\") "
Oct 30 19:51:54.890923 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:54.890901    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c7794b-a90d-46ca-9667-1729a711596e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c7794b-a90d-46ca-9667-1729a711596e-maas-api-openshift-ca\") pod \"27c7794b-a90d-46ca-9667-1729a711596e\" (UID: \"27c7794b-a90d-46ca-9667-1729a711596e\") "
Oct 30 19:51:54.891553 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:54.891528    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c7794b-a90d-46ca-9667-1729a711596e-maas-api-openshift-ca" pod "27c7794b-a90d-46ca-9667-1729a711596e" (UID: "27c7794b-a90d-46ca-9667-1729a711596e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:54.892999 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:54.892974    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c7794b-a90d-46ca-9667-1729a711596e-kube-api-access-vf4xp" pod "27c7794b-a90d-46ca-9667-1729a711596e" (UID: "27c7794b-a90d-46ca-9667-1729a711596e"). InnerVolumeSpecName "kube-api-access-vf4xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:54.892999 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:54.892981    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c7794b-a90d-46ca-9667-1729a711596e-maas-api-xks-ca" pod "27c7794b-a90d-46ca-9667-1729a711596e" (UID: "27c7794b-a90d-46ca-9667-1729a711596e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:54.992972 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:54.992937    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf4xp\" (UniqueName: \"kubernetes.io/projected/27c7794b-a90d-46ca-9667-1729a711596e-kube-api-access-vf4xp\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:54.992972 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:54.992973    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27c7794b-a90d-46ca-9667-1729a711596e-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:54.993233 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:54.992990    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27c7794b-a90d-46ca-9667-1729a711596e-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:55.245411 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.245376    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"]
Oct 30 19:51:55.245998 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.245983    4823 cpu_manager.go:414] "containerMap: removing container" podUID="27c7794b-a90d-46ca-9667-1729a711596e" containerName="revoke-keys"
Oct 30 19:51:55.246074 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.245999    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c7794b-a90d-46ca-9667-1729a711596e" containerName="revoke-keys"
Oct 30 19:51:55.246124 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.246115    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c7794b-a90d-46ca-9667-1729a711596e" containerName="revoke-keys"
Oct 30 19:51:55.249591 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.249571    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"
Oct 30 19:51:55.267802 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.267773    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"]
Oct 30 19:51:55.296607 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.296579    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99954dc3-5d88-47ec-9408-164decaf5451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e7be96c-xhbhf\" (UID: \"99954dc3-5d88-47ec-9408-164decaf5451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"
Oct 30 19:51:55.297125 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.297094    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99954dc3-5d88-47ec-9408-164decaf5451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e7be96c-xhbhf\" (UID: \"99954dc3-5d88-47ec-9408-164decaf5451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"
Oct 30 19:51:55.297269 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.297232    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5wgt\" (UniqueName: \"kubernetes.io/projected/99954dc3-5d88-47ec-9408-164decaf5451-kube-api-access-q5wgt\") pod \"maas-api-revoke-sub-4e7be96c-xhbhf\" (UID: \"99954dc3-5d88-47ec-9408-164decaf5451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"
Oct 30 19:51:55.398634 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.398591    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wgt\" (UniqueName: \"kubernetes.io/projected/99954dc3-5d88-47ec-9408-164decaf5451-kube-api-access-q5wgt\") pod \"maas-api-revoke-sub-4e7be96c-xhbhf\" (UID: \"99954dc3-5d88-47ec-9408-164decaf5451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"
Oct 30 19:51:55.398790 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.398718    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99954dc3-5d88-47ec-9408-164decaf5451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e7be96c-xhbhf\" (UID: \"99954dc3-5d88-47ec-9408-164decaf5451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"
Oct 30 19:51:55.398827 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.398798    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99954dc3-5d88-47ec-9408-164decaf5451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e7be96c-xhbhf\" (UID: \"99954dc3-5d88-47ec-9408-164decaf5451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"
Oct 30 19:51:55.399414 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.399395    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99954dc3-5d88-47ec-9408-164decaf5451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e7be96c-xhbhf\" (UID: \"99954dc3-5d88-47ec-9408-164decaf5451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"
Oct 30 19:51:55.401170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.401140    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99954dc3-5d88-47ec-9408-164decaf5451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e7be96c-xhbhf\" (UID: \"99954dc3-5d88-47ec-9408-164decaf5451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"
Oct 30 19:51:55.411802 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.411775    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wgt\" (UniqueName: \"kubernetes.io/projected/99954dc3-5d88-47ec-9408-164decaf5451-kube-api-access-q5wgt\") pod \"maas-api-revoke-sub-4e7be96c-xhbhf\" (UID: \"99954dc3-5d88-47ec-9408-164decaf5451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"
Oct 30 19:51:55.563235 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.563157    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"
Oct 30 19:51:55.721085 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.721052    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"
Oct 30 19:51:55.721486 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.721085    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5" event={"ID":"27c7794b-a90d-46ca-9667-1729a711596e","Type":"ContainerDied","Data":"bc376d476302ca6bd7daf18dc0a975a03065fce3443e2d189acc2ca51053f40b"}
Oct 30 19:51:55.721486 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.721142    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc376d476302ca6bd7daf18dc0a975a03065fce3443e2d189acc2ca51053f40b"
Oct 30 19:51:55.850139 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.850095    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"]
Oct 30 19:51:55.851302 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:51:55.851278    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99954dc3_5d88_47ec_9408_164decaf5451.slice/crio-78d6bed829303848bb0eae506a62d866b5b1a9d7f8d7fe171e5beebf0835813e WatchSource:0}: Error finding container 78d6bed829303848bb0eae506a62d866b5b1a9d7f8d7fe171e5beebf0835813e: Status 404 returned error can't find the container with id 78d6bed829303848bb0eae506a62d866b5b1a9d7f8d7fe171e5beebf0835813e
Oct 30 19:51:55.866487 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.866455    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"
Oct 30 19:51:55.925313 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.905554    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b647870-c146-4318-816a-ba78907e8b10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b647870-c146-4318-816a-ba78907e8b10-maas-api-openshift-ca\") pod \"5b647870-c146-4318-816a-ba78907e8b10\" (UID: \"5b647870-c146-4318-816a-ba78907e8b10\") "
Oct 30 19:51:55.925313 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.905637    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b647870-c146-4318-816a-ba78907e8b10-kube-api-access-r42j7\" (UniqueName: \"kubernetes.io/projected/5b647870-c146-4318-816a-ba78907e8b10-kube-api-access-r42j7\") pod \"5b647870-c146-4318-816a-ba78907e8b10\" (UID: \"5b647870-c146-4318-816a-ba78907e8b10\") "
Oct 30 19:51:55.925313 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.905674    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b647870-c146-4318-816a-ba78907e8b10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b647870-c146-4318-816a-ba78907e8b10-maas-api-xks-ca\") pod \"5b647870-c146-4318-816a-ba78907e8b10\" (UID: \"5b647870-c146-4318-816a-ba78907e8b10\") "
Oct 30 19:51:55.925313 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.907917    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b647870-c146-4318-816a-ba78907e8b10-maas-api-openshift-ca" pod "5b647870-c146-4318-816a-ba78907e8b10" (UID: "5b647870-c146-4318-816a-ba78907e8b10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:55.944876 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.933393    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b647870-c146-4318-816a-ba78907e8b10-maas-api-xks-ca" pod "5b647870-c146-4318-816a-ba78907e8b10" (UID: "5b647870-c146-4318-816a-ba78907e8b10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:55.944876 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:55.936761    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b647870-c146-4318-816a-ba78907e8b10-kube-api-access-r42j7" pod "5b647870-c146-4318-816a-ba78907e8b10" (UID: "5b647870-c146-4318-816a-ba78907e8b10"). InnerVolumeSpecName "kube-api-access-r42j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:56.007686 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.007654    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b647870-c146-4318-816a-ba78907e8b10-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.007686 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.007684    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r42j7\" (UniqueName: \"kubernetes.io/projected/5b647870-c146-4318-816a-ba78907e8b10-kube-api-access-r42j7\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.007900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.007695    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b647870-c146-4318-816a-ba78907e8b10-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.662509 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.662474    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"]
Oct 30 19:51:56.663146 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.663126    4823 cpu_manager.go:414] "containerMap: removing container" podUID="5b647870-c146-4318-816a-ba78907e8b10" containerName="revoke-keys"
Oct 30 19:51:56.663247 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.663147    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b647870-c146-4318-816a-ba78907e8b10" containerName="revoke-keys"
Oct 30 19:51:56.663344 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.663331    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b647870-c146-4318-816a-ba78907e8b10" containerName="revoke-keys"
Oct 30 19:51:56.667010 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.666990    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"
Oct 30 19:51:56.729093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.729049    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf" event={"ID":"99954dc3-5d88-47ec-9408-164decaf5451","Type":"ContainerStarted","Data":"f84ef62429fd5defca0d215c407ee44efd8fd4c3e0471e9e433a8c0cb0523742"}
Oct 30 19:51:56.729529 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.729111    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf" event={"ID":"99954dc3-5d88-47ec-9408-164decaf5451","Type":"ContainerStarted","Data":"78d6bed829303848bb0eae506a62d866b5b1a9d7f8d7fe171e5beebf0835813e"}
Oct 30 19:51:56.730849 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.730814    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz" event={"ID":"5b647870-c146-4318-816a-ba78907e8b10","Type":"ContainerDied","Data":"f4fd5524bb90a32b419c288c314c33a29e2ac3c146da2b459c9336d9d5e0c850"}
Oct 30 19:51:56.731000 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.730882    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4fd5524bb90a32b419c288c314c33a29e2ac3c146da2b459c9336d9d5e0c850"
Oct 30 19:51:56.731000 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.730887    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"
Oct 30 19:51:56.731596 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.731571    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2svjx\" (UniqueName: \"kubernetes.io/projected/bc7b9f48-eab6-4e33-ac21-66265f054d57-kube-api-access-2svjx\") pod \"maas-api-revoke-sub-db4410c9-6f97m\" (UID: \"bc7b9f48-eab6-4e33-ac21-66265f054d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"
Oct 30 19:51:56.731786 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.731750    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7b9f48-eab6-4e33-ac21-66265f054d57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db4410c9-6f97m\" (UID: \"bc7b9f48-eab6-4e33-ac21-66265f054d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"
Oct 30 19:51:56.731919 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.731835    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7b9f48-eab6-4e33-ac21-66265f054d57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db4410c9-6f97m\" (UID: \"bc7b9f48-eab6-4e33-ac21-66265f054d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"
Oct 30 19:51:56.736002 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.735952    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"]
Oct 30 19:51:56.822728 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.822678    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf" podStartSLOduration=1.822663426 podStartE2EDuration="1.822663426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:56.821728364 +0000 UTC m=+1316.714342691" watchObservedRunningTime="2026-10-30 19:51:56.822663426 +0000 UTC m=+1316.715277750"
Oct 30 19:51:56.833560 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.833531    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2svjx\" (UniqueName: \"kubernetes.io/projected/bc7b9f48-eab6-4e33-ac21-66265f054d57-kube-api-access-2svjx\") pod \"maas-api-revoke-sub-db4410c9-6f97m\" (UID: \"bc7b9f48-eab6-4e33-ac21-66265f054d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"
Oct 30 19:51:56.833672 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.833565    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7b9f48-eab6-4e33-ac21-66265f054d57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db4410c9-6f97m\" (UID: \"bc7b9f48-eab6-4e33-ac21-66265f054d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"
Oct 30 19:51:56.833706 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.833682    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7b9f48-eab6-4e33-ac21-66265f054d57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db4410c9-6f97m\" (UID: \"bc7b9f48-eab6-4e33-ac21-66265f054d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"
Oct 30 19:51:56.834204 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.834183    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7b9f48-eab6-4e33-ac21-66265f054d57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db4410c9-6f97m\" (UID: \"bc7b9f48-eab6-4e33-ac21-66265f054d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"
Oct 30 19:51:56.836002 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.835982    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7b9f48-eab6-4e33-ac21-66265f054d57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db4410c9-6f97m\" (UID: \"bc7b9f48-eab6-4e33-ac21-66265f054d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"
Oct 30 19:51:56.889579 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.889552    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2svjx\" (UniqueName: \"kubernetes.io/projected/bc7b9f48-eab6-4e33-ac21-66265f054d57-kube-api-access-2svjx\") pod \"maas-api-revoke-sub-db4410c9-6f97m\" (UID: \"bc7b9f48-eab6-4e33-ac21-66265f054d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"
Oct 30 19:51:56.980383 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:56.980307    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"
Oct 30 19:51:57.282016 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:57.281988    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"]
Oct 30 19:51:57.283487 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:51:57.283460    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7b9f48_eab6_4e33_ac21_66265f054d57.slice/crio-7457fdd242a16a86c21ad4b121e42421fb1787f86dc409e9d27f1b9790ceac39 WatchSource:0}: Error finding container 7457fdd242a16a86c21ad4b121e42421fb1787f86dc409e9d27f1b9790ceac39: Status 404 returned error can't find the container with id 7457fdd242a16a86c21ad4b121e42421fb1787f86dc409e9d27f1b9790ceac39
Oct 30 19:51:57.739084 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:57.739049    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m" event={"ID":"bc7b9f48-eab6-4e33-ac21-66265f054d57","Type":"ContainerStarted","Data":"47a66079d46d0d1cd904b752a7a0fa1204875f327f15df03ed025a29d858b7af"}
Oct 30 19:51:57.739505 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:57.739105    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m" event={"ID":"bc7b9f48-eab6-4e33-ac21-66265f054d57","Type":"ContainerStarted","Data":"7457fdd242a16a86c21ad4b121e42421fb1787f86dc409e9d27f1b9790ceac39"}
Oct 30 19:51:57.740720 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:57.740697    4823 generic.go:356] "Generic (PLEG): container finished" podID="99954dc3-5d88-47ec-9408-164decaf5451" containerID="f84ef62429fd5defca0d215c407ee44efd8fd4c3e0471e9e433a8c0cb0523742" exitCode=0
Oct 30 19:51:57.740828 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:57.740782    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf" event={"ID":"99954dc3-5d88-47ec-9408-164decaf5451","Type":"ContainerDied","Data":"f84ef62429fd5defca0d215c407ee44efd8fd4c3e0471e9e433a8c0cb0523742"}
Oct 30 19:51:57.776929 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:57.776857    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m" podStartSLOduration=1.77684376 podStartE2EDuration="1.77684376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:57.77328892 +0000 UTC m=+1317.665903244" watchObservedRunningTime="2026-10-30 19:51:57.77684376 +0000 UTC m=+1317.669458084"
Oct 30 19:51:58.748336 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:58.748298    4823 generic.go:356] "Generic (PLEG): container finished" podID="bc7b9f48-eab6-4e33-ac21-66265f054d57" containerID="47a66079d46d0d1cd904b752a7a0fa1204875f327f15df03ed025a29d858b7af" exitCode=0
Oct 30 19:51:58.748702 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:58.748380    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m" event={"ID":"bc7b9f48-eab6-4e33-ac21-66265f054d57","Type":"ContainerDied","Data":"47a66079d46d0d1cd904b752a7a0fa1204875f327f15df03ed025a29d858b7af"}
Oct 30 19:51:58.934949 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:58.934918    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"
Oct 30 19:51:59.066544 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.066472    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99954dc3-5d88-47ec-9408-164decaf5451-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99954dc3-5d88-47ec-9408-164decaf5451-maas-api-xks-ca\") pod \"99954dc3-5d88-47ec-9408-164decaf5451\" (UID: \"99954dc3-5d88-47ec-9408-164decaf5451\") "
Oct 30 19:51:59.066695 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.066636    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99954dc3-5d88-47ec-9408-164decaf5451-kube-api-access-q5wgt\" (UniqueName: \"kubernetes.io/projected/99954dc3-5d88-47ec-9408-164decaf5451-kube-api-access-q5wgt\") pod \"99954dc3-5d88-47ec-9408-164decaf5451\" (UID: \"99954dc3-5d88-47ec-9408-164decaf5451\") "
Oct 30 19:51:59.066695 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.066690    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99954dc3-5d88-47ec-9408-164decaf5451-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99954dc3-5d88-47ec-9408-164decaf5451-maas-api-openshift-ca\") pod \"99954dc3-5d88-47ec-9408-164decaf5451\" (UID: \"99954dc3-5d88-47ec-9408-164decaf5451\") "
Oct 30 19:51:59.067087 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.067059    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99954dc3-5d88-47ec-9408-164decaf5451-maas-api-openshift-ca" pod "99954dc3-5d88-47ec-9408-164decaf5451" (UID: "99954dc3-5d88-47ec-9408-164decaf5451"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:59.068625 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.068598    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99954dc3-5d88-47ec-9408-164decaf5451-maas-api-xks-ca" pod "99954dc3-5d88-47ec-9408-164decaf5451" (UID: "99954dc3-5d88-47ec-9408-164decaf5451"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:59.068806 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.068781    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99954dc3-5d88-47ec-9408-164decaf5451-kube-api-access-q5wgt" pod "99954dc3-5d88-47ec-9408-164decaf5451" (UID: "99954dc3-5d88-47ec-9408-164decaf5451"). InnerVolumeSpecName "kube-api-access-q5wgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:59.168484 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.168446    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99954dc3-5d88-47ec-9408-164decaf5451-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:59.168484 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.168479    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5wgt\" (UniqueName: \"kubernetes.io/projected/99954dc3-5d88-47ec-9408-164decaf5451-kube-api-access-q5wgt\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:59.168484 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.168490    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99954dc3-5d88-47ec-9408-164decaf5451-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:59.756140 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.756108    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"
Oct 30 19:51:59.756635 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.756141    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf" event={"ID":"99954dc3-5d88-47ec-9408-164decaf5451","Type":"ContainerDied","Data":"78d6bed829303848bb0eae506a62d866b5b1a9d7f8d7fe171e5beebf0835813e"}
Oct 30 19:51:59.756635 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.756188    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78d6bed829303848bb0eae506a62d866b5b1a9d7f8d7fe171e5beebf0835813e"
Oct 30 19:51:59.926676 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.926651    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"
Oct 30 19:51:59.978836 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.978805    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7b9f48-eab6-4e33-ac21-66265f054d57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7b9f48-eab6-4e33-ac21-66265f054d57-maas-api-openshift-ca\") pod \"bc7b9f48-eab6-4e33-ac21-66265f054d57\" (UID: \"bc7b9f48-eab6-4e33-ac21-66265f054d57\") "
Oct 30 19:51:59.979014 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.978931    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7b9f48-eab6-4e33-ac21-66265f054d57-kube-api-access-2svjx\" (UniqueName: \"kubernetes.io/projected/bc7b9f48-eab6-4e33-ac21-66265f054d57-kube-api-access-2svjx\") pod \"bc7b9f48-eab6-4e33-ac21-66265f054d57\" (UID: \"bc7b9f48-eab6-4e33-ac21-66265f054d57\") "
Oct 30 19:51:59.979133 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.979111    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7b9f48-eab6-4e33-ac21-66265f054d57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7b9f48-eab6-4e33-ac21-66265f054d57-maas-api-xks-ca\") pod \"bc7b9f48-eab6-4e33-ac21-66265f054d57\" (UID: \"bc7b9f48-eab6-4e33-ac21-66265f054d57\") "
Oct 30 19:51:59.979300 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.979275    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7b9f48-eab6-4e33-ac21-66265f054d57-maas-api-openshift-ca" pod "bc7b9f48-eab6-4e33-ac21-66265f054d57" (UID: "bc7b9f48-eab6-4e33-ac21-66265f054d57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:59.979590 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.979569    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7b9f48-eab6-4e33-ac21-66265f054d57-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:59.982036 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.981530    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7b9f48-eab6-4e33-ac21-66265f054d57-kube-api-access-2svjx" pod "bc7b9f48-eab6-4e33-ac21-66265f054d57" (UID: "bc7b9f48-eab6-4e33-ac21-66265f054d57"). InnerVolumeSpecName "kube-api-access-2svjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:59.982036 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:51:59.981619    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7b9f48-eab6-4e33-ac21-66265f054d57-maas-api-xks-ca" pod "bc7b9f48-eab6-4e33-ac21-66265f054d57" (UID: "bc7b9f48-eab6-4e33-ac21-66265f054d57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:00.080705 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:00.080621    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7b9f48-eab6-4e33-ac21-66265f054d57-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:00.080705 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:00.080654    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2svjx\" (UniqueName: \"kubernetes.io/projected/bc7b9f48-eab6-4e33-ac21-66265f054d57-kube-api-access-2svjx\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:00.764754 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:00.764724    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"
Oct 30 19:52:00.764754 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:00.764737    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m" event={"ID":"bc7b9f48-eab6-4e33-ac21-66265f054d57","Type":"ContainerDied","Data":"7457fdd242a16a86c21ad4b121e42421fb1787f86dc409e9d27f1b9790ceac39"}
Oct 30 19:52:00.765227 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:00.764828    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7457fdd242a16a86c21ad4b121e42421fb1787f86dc409e9d27f1b9790ceac39"
Oct 30 19:52:02.015045 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:52:02.015004    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:52:02.032294 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:02.032268    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:52:05.207429 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.207394    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"]
Oct 30 19:52:05.208227 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.208205    4823 cpu_manager.go:414] "containerMap: removing container" podUID="99954dc3-5d88-47ec-9408-164decaf5451" containerName="revoke-keys"
Oct 30 19:52:05.208227 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.208228    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99954dc3-5d88-47ec-9408-164decaf5451" containerName="revoke-keys"
Oct 30 19:52:05.208382 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.208239    4823 cpu_manager.go:414] "containerMap: removing container" podUID="bc7b9f48-eab6-4e33-ac21-66265f054d57" containerName="revoke-keys"
Oct 30 19:52:05.208382 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.208247    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7b9f48-eab6-4e33-ac21-66265f054d57" containerName="revoke-keys"
Oct 30 19:52:05.208481 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.208433    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7b9f48-eab6-4e33-ac21-66265f054d57" containerName="revoke-keys"
Oct 30 19:52:05.208481 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.208448    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="99954dc3-5d88-47ec-9408-164decaf5451" containerName="revoke-keys"
Oct 30 19:52:05.214422 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.214400    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"
Oct 30 19:52:05.223591 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.223566    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:52:05.223723 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.223633    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:52:05.226431 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.226404    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"]
Oct 30 19:52:05.226812 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.226790    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:52:05.226947 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.226835    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:52:05.283506 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.283462    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgkxg\" (UniqueName: \"kubernetes.io/projected/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-kube-api-access-cgkxg\") pod \"maas-api-revoke-sub-d12a9b3f-sjqff\" (UID: \"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"
Oct 30 19:52:05.283671 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.283621    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d12a9b3f-sjqff\" (UID: \"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"
Oct 30 19:52:05.283671 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.283655    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d12a9b3f-sjqff\" (UID: \"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"
Oct 30 19:52:05.384696 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.384657    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgkxg\" (UniqueName: \"kubernetes.io/projected/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-kube-api-access-cgkxg\") pod \"maas-api-revoke-sub-d12a9b3f-sjqff\" (UID: \"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"
Oct 30 19:52:05.384910 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.384770    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d12a9b3f-sjqff\" (UID: \"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"
Oct 30 19:52:05.384910 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.384804    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d12a9b3f-sjqff\" (UID: \"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"
Oct 30 19:52:05.385462 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.385443    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d12a9b3f-sjqff\" (UID: \"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"
Oct 30 19:52:05.387275 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.387251    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d12a9b3f-sjqff\" (UID: \"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"
Oct 30 19:52:05.397732 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.397704    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgkxg\" (UniqueName: \"kubernetes.io/projected/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-kube-api-access-cgkxg\") pod \"maas-api-revoke-sub-d12a9b3f-sjqff\" (UID: \"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"
Oct 30 19:52:05.531990 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.531910    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"
Oct 30 19:52:05.824542 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:05.824513    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"]
Oct 30 19:52:05.826168 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:52:05.826133    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec1e317_b121_4b97_bb23_9ee5ab2b2eb6.slice/crio-52709e5d40ce891927ada2cf9dd13d0b65a6fb958db760cb4ea4a59de83c4fba WatchSource:0}: Error finding container 52709e5d40ce891927ada2cf9dd13d0b65a6fb958db760cb4ea4a59de83c4fba: Status 404 returned error can't find the container with id 52709e5d40ce891927ada2cf9dd13d0b65a6fb958db760cb4ea4a59de83c4fba
Oct 30 19:52:06.833854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:06.833823    4823 generic.go:356] "Generic (PLEG): container finished" podID="4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6" containerID="e371d229d89f18cfd51e883321a39e606bf40b22c77df3532de60c015d82a074" exitCode=0
Oct 30 19:52:06.834228 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:06.833914    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff" event={"ID":"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6","Type":"ContainerDied","Data":"e371d229d89f18cfd51e883321a39e606bf40b22c77df3532de60c015d82a074"}
Oct 30 19:52:06.834228 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:06.833965    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff" event={"ID":"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6","Type":"ContainerStarted","Data":"52709e5d40ce891927ada2cf9dd13d0b65a6fb958db760cb4ea4a59de83c4fba"}
Oct 30 19:52:06.860901 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:06.860848    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"]
Oct 30 19:52:06.865329 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:06.865305    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"
Oct 30 19:52:06.924918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:06.924886    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"]
Oct 30 19:52:07.005577 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.005539    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76698bf7-4947-4e62-8bda-b4e2a17c44f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f54eed2-jwdqh\" (UID: \"76698bf7-4947-4e62-8bda-b4e2a17c44f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"
Oct 30 19:52:07.005770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.005615    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jn59\" (UniqueName: \"kubernetes.io/projected/76698bf7-4947-4e62-8bda-b4e2a17c44f5-kube-api-access-6jn59\") pod \"maas-api-revoke-sub-7f54eed2-jwdqh\" (UID: \"76698bf7-4947-4e62-8bda-b4e2a17c44f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"
Oct 30 19:52:07.005770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.005740    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76698bf7-4947-4e62-8bda-b4e2a17c44f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f54eed2-jwdqh\" (UID: \"76698bf7-4947-4e62-8bda-b4e2a17c44f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"
Oct 30 19:52:07.030328 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.030243    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"]
Oct 30 19:52:07.034741 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.034719    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"
Oct 30 19:52:07.045912 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.045885    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"]
Oct 30 19:52:07.106506 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.106473    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76698bf7-4947-4e62-8bda-b4e2a17c44f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f54eed2-jwdqh\" (UID: \"76698bf7-4947-4e62-8bda-b4e2a17c44f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"
Oct 30 19:52:07.106680 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.106537    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc739f79-1809-42c5-8660-816cfa106a70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-033834de-czh2l\" (UID: \"cc739f79-1809-42c5-8660-816cfa106a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"
Oct 30 19:52:07.106680 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.106580    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bzd7\" (UniqueName: \"kubernetes.io/projected/cc739f79-1809-42c5-8660-816cfa106a70-kube-api-access-7bzd7\") pod \"maas-api-revoke-sub-033834de-czh2l\" (UID: \"cc739f79-1809-42c5-8660-816cfa106a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"
Oct 30 19:52:07.106810 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.106687    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc739f79-1809-42c5-8660-816cfa106a70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-033834de-czh2l\" (UID: \"cc739f79-1809-42c5-8660-816cfa106a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"
Oct 30 19:52:07.106893 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.106807    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76698bf7-4947-4e62-8bda-b4e2a17c44f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f54eed2-jwdqh\" (UID: \"76698bf7-4947-4e62-8bda-b4e2a17c44f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"
Oct 30 19:52:07.106972 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.106954    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jn59\" (UniqueName: \"kubernetes.io/projected/76698bf7-4947-4e62-8bda-b4e2a17c44f5-kube-api-access-6jn59\") pod \"maas-api-revoke-sub-7f54eed2-jwdqh\" (UID: \"76698bf7-4947-4e62-8bda-b4e2a17c44f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"
Oct 30 19:52:07.107463 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.107440    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76698bf7-4947-4e62-8bda-b4e2a17c44f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f54eed2-jwdqh\" (UID: \"76698bf7-4947-4e62-8bda-b4e2a17c44f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"
Oct 30 19:52:07.109045 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.109025    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76698bf7-4947-4e62-8bda-b4e2a17c44f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f54eed2-jwdqh\" (UID: \"76698bf7-4947-4e62-8bda-b4e2a17c44f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"
Oct 30 19:52:07.119486 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.119457    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jn59\" (UniqueName: \"kubernetes.io/projected/76698bf7-4947-4e62-8bda-b4e2a17c44f5-kube-api-access-6jn59\") pod \"maas-api-revoke-sub-7f54eed2-jwdqh\" (UID: \"76698bf7-4947-4e62-8bda-b4e2a17c44f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"
Oct 30 19:52:07.182543 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.182509    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"
Oct 30 19:52:07.207797 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.207770    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc739f79-1809-42c5-8660-816cfa106a70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-033834de-czh2l\" (UID: \"cc739f79-1809-42c5-8660-816cfa106a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"
Oct 30 19:52:07.207956 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.207899    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bzd7\" (UniqueName: \"kubernetes.io/projected/cc739f79-1809-42c5-8660-816cfa106a70-kube-api-access-7bzd7\") pod \"maas-api-revoke-sub-033834de-czh2l\" (UID: \"cc739f79-1809-42c5-8660-816cfa106a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"
Oct 30 19:52:07.208021 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.207984    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc739f79-1809-42c5-8660-816cfa106a70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-033834de-czh2l\" (UID: \"cc739f79-1809-42c5-8660-816cfa106a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"
Oct 30 19:52:07.208589 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.208564    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc739f79-1809-42c5-8660-816cfa106a70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-033834de-czh2l\" (UID: \"cc739f79-1809-42c5-8660-816cfa106a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"
Oct 30 19:52:07.210422 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.210377    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc739f79-1809-42c5-8660-816cfa106a70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-033834de-czh2l\" (UID: \"cc739f79-1809-42c5-8660-816cfa106a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"
Oct 30 19:52:07.219461 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.219439    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bzd7\" (UniqueName: \"kubernetes.io/projected/cc739f79-1809-42c5-8660-816cfa106a70-kube-api-access-7bzd7\") pod \"maas-api-revoke-sub-033834de-czh2l\" (UID: \"cc739f79-1809-42c5-8660-816cfa106a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"
Oct 30 19:52:07.350139 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.350058    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"
Oct 30 19:52:07.482180 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.480174    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"]
Oct 30 19:52:07.482180 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:52:07.481454    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76698bf7_4947_4e62_8bda_b4e2a17c44f5.slice/crio-5abffb40e1a18a780ee454627b4e202869b8e3eea3c68cdaad32206159f6412c WatchSource:0}: Error finding container 5abffb40e1a18a780ee454627b4e202869b8e3eea3c68cdaad32206159f6412c: Status 404 returned error can't find the container with id 5abffb40e1a18a780ee454627b4e202869b8e3eea3c68cdaad32206159f6412c
Oct 30 19:52:07.650629 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.650605    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"]
Oct 30 19:52:07.653066 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:52:07.653032    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc739f79_1809_42c5_8660_816cfa106a70.slice/crio-15b16276dabf05c317c585bbc89b20ab57e7d51c84ba66870b187ee6c79ec15c WatchSource:0}: Error finding container 15b16276dabf05c317c585bbc89b20ab57e7d51c84ba66870b187ee6c79ec15c: Status 404 returned error can't find the container with id 15b16276dabf05c317c585bbc89b20ab57e7d51c84ba66870b187ee6c79ec15c
Oct 30 19:52:07.841747 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.841707    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l" event={"ID":"cc739f79-1809-42c5-8660-816cfa106a70","Type":"ContainerStarted","Data":"15b16276dabf05c317c585bbc89b20ab57e7d51c84ba66870b187ee6c79ec15c"}
Oct 30 19:52:07.843309 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.843283    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh" event={"ID":"76698bf7-4947-4e62-8bda-b4e2a17c44f5","Type":"ContainerStarted","Data":"9054dfe8c7832ce80e14f85a963de58d9d1be519f9a891c5cc7efee59a2f89c2"}
Oct 30 19:52:07.843396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.843330    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh" event={"ID":"76698bf7-4947-4e62-8bda-b4e2a17c44f5","Type":"ContainerStarted","Data":"5abffb40e1a18a780ee454627b4e202869b8e3eea3c68cdaad32206159f6412c"}
Oct 30 19:52:07.871012 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.870953    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh" podStartSLOduration=1.870934943 podStartE2EDuration="1.870934943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:07.867401042 +0000 UTC m=+1327.760015367" watchObservedRunningTime="2026-10-30 19:52:07.870934943 +0000 UTC m=+1327.763549268"
Oct 30 19:52:07.996488 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:07.996461    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"
Oct 30 19:52:08.121229 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.121202    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-kube-api-access-cgkxg\" (UniqueName: \"kubernetes.io/projected/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-kube-api-access-cgkxg\") pod \"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6\" (UID: \"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6\") "
Oct 30 19:52:08.121360 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.121246    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-maas-api-openshift-ca\") pod \"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6\" (UID: \"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6\") "
Oct 30 19:52:08.121360 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.121268    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-maas-api-xks-ca\") pod \"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6\" (UID: \"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6\") "
Oct 30 19:52:08.121604 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.121581    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-maas-api-openshift-ca" pod "4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6" (UID: "4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:08.123404 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.123380    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-maas-api-xks-ca" pod "4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6" (UID: "4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:08.123526 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.123482    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-kube-api-access-cgkxg" pod "4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6" (UID: "4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6"). InnerVolumeSpecName "kube-api-access-cgkxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:08.222967 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.222851    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgkxg\" (UniqueName: \"kubernetes.io/projected/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-kube-api-access-cgkxg\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:08.222967 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.222910    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:08.222967 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.222921    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:08.531077 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.531003    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"]
Oct 30 19:52:08.531600 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.531586    4823 cpu_manager.go:414] "containerMap: removing container" podUID="4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6" containerName="revoke-keys"
Oct 30 19:52:08.531645 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.531602    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6" containerName="revoke-keys"
Oct 30 19:52:08.531727 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.531718    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6" containerName="revoke-keys"
Oct 30 19:52:08.535033 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.535017    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"
Oct 30 19:52:08.556964 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.556933    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"]
Oct 30 19:52:08.627745 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.627715    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47a7ab5b-4phst\" (UID: \"a1583ba0-2c91-4348-bfd0-83e2b43bac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"
Oct 30 19:52:08.627925 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.627781    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5n55\" (UniqueName: \"kubernetes.io/projected/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-kube-api-access-k5n55\") pod \"maas-api-revoke-sub-47a7ab5b-4phst\" (UID: \"a1583ba0-2c91-4348-bfd0-83e2b43bac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"
Oct 30 19:52:08.627925 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.627907    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47a7ab5b-4phst\" (UID: \"a1583ba0-2c91-4348-bfd0-83e2b43bac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"
Oct 30 19:52:08.729631 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.729598    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47a7ab5b-4phst\" (UID: \"a1583ba0-2c91-4348-bfd0-83e2b43bac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"
Oct 30 19:52:08.729810 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.729669    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5n55\" (UniqueName: \"kubernetes.io/projected/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-kube-api-access-k5n55\") pod \"maas-api-revoke-sub-47a7ab5b-4phst\" (UID: \"a1583ba0-2c91-4348-bfd0-83e2b43bac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"
Oct 30 19:52:08.729810 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.729714    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47a7ab5b-4phst\" (UID: \"a1583ba0-2c91-4348-bfd0-83e2b43bac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"
Oct 30 19:52:08.730485 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.730460    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47a7ab5b-4phst\" (UID: \"a1583ba0-2c91-4348-bfd0-83e2b43bac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"
Oct 30 19:52:08.732053 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.732028    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47a7ab5b-4phst\" (UID: \"a1583ba0-2c91-4348-bfd0-83e2b43bac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"
Oct 30 19:52:08.755733 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.755689    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5n55\" (UniqueName: \"kubernetes.io/projected/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-kube-api-access-k5n55\") pod \"maas-api-revoke-sub-47a7ab5b-4phst\" (UID: \"a1583ba0-2c91-4348-bfd0-83e2b43bac2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"
Oct 30 19:52:08.848248 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.848165    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"
Oct 30 19:52:08.851755 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.851499    4823 generic.go:356] "Generic (PLEG): container finished" podID="cc739f79-1809-42c5-8660-816cfa106a70" containerID="ec2fcd056bed347ab93a4cc248f466b440dfca06a8d0e60d0eec97fe0ae4b5e4" exitCode=0
Oct 30 19:52:08.851755 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.851603    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l" event={"ID":"cc739f79-1809-42c5-8660-816cfa106a70","Type":"ContainerDied","Data":"ec2fcd056bed347ab93a4cc248f466b440dfca06a8d0e60d0eec97fe0ae4b5e4"}
Oct 30 19:52:08.853408 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.853380    4823 generic.go:356] "Generic (PLEG): container finished" podID="76698bf7-4947-4e62-8bda-b4e2a17c44f5" containerID="9054dfe8c7832ce80e14f85a963de58d9d1be519f9a891c5cc7efee59a2f89c2" exitCode=0
Oct 30 19:52:08.853522 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.853461    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh" event={"ID":"76698bf7-4947-4e62-8bda-b4e2a17c44f5","Type":"ContainerDied","Data":"9054dfe8c7832ce80e14f85a963de58d9d1be519f9a891c5cc7efee59a2f89c2"}
Oct 30 19:52:08.854795 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.854779    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"
Oct 30 19:52:08.854898 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.854826    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff" event={"ID":"4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6","Type":"ContainerDied","Data":"52709e5d40ce891927ada2cf9dd13d0b65a6fb958db760cb4ea4a59de83c4fba"}
Oct 30 19:52:08.854898 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:08.854893    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52709e5d40ce891927ada2cf9dd13d0b65a6fb958db760cb4ea4a59de83c4fba"
Oct 30 19:52:09.146247 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:09.146216    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"]
Oct 30 19:52:09.147590 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:52:09.147567    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1583ba0_2c91_4348_bfd0_83e2b43bac2c.slice/crio-ac928aed694c55e103bf16a98b24e431728e6608f7c63c23abc84cd08a3fb95b WatchSource:0}: Error finding container ac928aed694c55e103bf16a98b24e431728e6608f7c63c23abc84cd08a3fb95b: Status 404 returned error can't find the container with id ac928aed694c55e103bf16a98b24e431728e6608f7c63c23abc84cd08a3fb95b
Oct 30 19:52:09.864135 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:09.864103    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst" event={"ID":"a1583ba0-2c91-4348-bfd0-83e2b43bac2c","Type":"ContainerStarted","Data":"0c58fede203767ba8bc9bdd8f109150df795dc63f1482605d1b9eb99911007a6"}
Oct 30 19:52:09.864565 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:09.864157    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst" event={"ID":"a1583ba0-2c91-4348-bfd0-83e2b43bac2c","Type":"ContainerStarted","Data":"ac928aed694c55e103bf16a98b24e431728e6608f7c63c23abc84cd08a3fb95b"}
Oct 30 19:52:09.892283 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:09.892224    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst" podStartSLOduration=1.89220672 podStartE2EDuration="1.89220672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:09.887132722 +0000 UTC m=+1329.779747047" watchObservedRunningTime="2026-10-30 19:52:09.89220672 +0000 UTC m=+1329.784821043"
Oct 30 19:52:10.087530 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.087501    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"
Oct 30 19:52:10.099847 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.099821    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"
Oct 30 19:52:10.247029 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.246954    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc739f79-1809-42c5-8660-816cfa106a70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc739f79-1809-42c5-8660-816cfa106a70-maas-api-xks-ca\") pod \"cc739f79-1809-42c5-8660-816cfa106a70\" (UID: \"cc739f79-1809-42c5-8660-816cfa106a70\") "
Oct 30 19:52:10.247029 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.246993    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc739f79-1809-42c5-8660-816cfa106a70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc739f79-1809-42c5-8660-816cfa106a70-maas-api-openshift-ca\") pod \"cc739f79-1809-42c5-8660-816cfa106a70\" (UID: \"cc739f79-1809-42c5-8660-816cfa106a70\") "
Oct 30 19:52:10.247219 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.247118    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76698bf7-4947-4e62-8bda-b4e2a17c44f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76698bf7-4947-4e62-8bda-b4e2a17c44f5-maas-api-openshift-ca\") pod \"76698bf7-4947-4e62-8bda-b4e2a17c44f5\" (UID: \"76698bf7-4947-4e62-8bda-b4e2a17c44f5\") "
Oct 30 19:52:10.247219 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.247164    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76698bf7-4947-4e62-8bda-b4e2a17c44f5-kube-api-access-6jn59\" (UniqueName: \"kubernetes.io/projected/76698bf7-4947-4e62-8bda-b4e2a17c44f5-kube-api-access-6jn59\") pod \"76698bf7-4947-4e62-8bda-b4e2a17c44f5\" (UID: \"76698bf7-4947-4e62-8bda-b4e2a17c44f5\") "
Oct 30 19:52:10.247314 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.247295    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc739f79-1809-42c5-8660-816cfa106a70-kube-api-access-7bzd7\" (UniqueName: \"kubernetes.io/projected/cc739f79-1809-42c5-8660-816cfa106a70-kube-api-access-7bzd7\") pod \"cc739f79-1809-42c5-8660-816cfa106a70\" (UID: \"cc739f79-1809-42c5-8660-816cfa106a70\") "
Oct 30 19:52:10.247414 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.247387    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc739f79-1809-42c5-8660-816cfa106a70-maas-api-openshift-ca" pod "cc739f79-1809-42c5-8660-816cfa106a70" (UID: "cc739f79-1809-42c5-8660-816cfa106a70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:10.247501 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.247398    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76698bf7-4947-4e62-8bda-b4e2a17c44f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76698bf7-4947-4e62-8bda-b4e2a17c44f5-maas-api-xks-ca\") pod \"76698bf7-4947-4e62-8bda-b4e2a17c44f5\" (UID: \"76698bf7-4947-4e62-8bda-b4e2a17c44f5\") "
Oct 30 19:52:10.247501 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.247488    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76698bf7-4947-4e62-8bda-b4e2a17c44f5-maas-api-openshift-ca" pod "76698bf7-4947-4e62-8bda-b4e2a17c44f5" (UID: "76698bf7-4947-4e62-8bda-b4e2a17c44f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:10.247971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.247950    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc739f79-1809-42c5-8660-816cfa106a70-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:10.247971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.247971    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76698bf7-4947-4e62-8bda-b4e2a17c44f5-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:10.249397 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.249376    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76698bf7-4947-4e62-8bda-b4e2a17c44f5-maas-api-xks-ca" pod "76698bf7-4947-4e62-8bda-b4e2a17c44f5" (UID: "76698bf7-4947-4e62-8bda-b4e2a17c44f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:10.249397 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.249390    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc739f79-1809-42c5-8660-816cfa106a70-kube-api-access-7bzd7" pod "cc739f79-1809-42c5-8660-816cfa106a70" (UID: "cc739f79-1809-42c5-8660-816cfa106a70"). InnerVolumeSpecName "kube-api-access-7bzd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:10.249539 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.249488    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc739f79-1809-42c5-8660-816cfa106a70-maas-api-xks-ca" pod "cc739f79-1809-42c5-8660-816cfa106a70" (UID: "cc739f79-1809-42c5-8660-816cfa106a70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:10.249539 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.249528    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76698bf7-4947-4e62-8bda-b4e2a17c44f5-kube-api-access-6jn59" pod "76698bf7-4947-4e62-8bda-b4e2a17c44f5" (UID: "76698bf7-4947-4e62-8bda-b4e2a17c44f5"). InnerVolumeSpecName "kube-api-access-6jn59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:10.349658 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.349628    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jn59\" (UniqueName: \"kubernetes.io/projected/76698bf7-4947-4e62-8bda-b4e2a17c44f5-kube-api-access-6jn59\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:10.349658 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.349655    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bzd7\" (UniqueName: \"kubernetes.io/projected/cc739f79-1809-42c5-8660-816cfa106a70-kube-api-access-7bzd7\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:10.349658 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.349664    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76698bf7-4947-4e62-8bda-b4e2a17c44f5-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:10.349852 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.349674    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc739f79-1809-42c5-8660-816cfa106a70-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:10.873847 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.873811    4823 generic.go:356] "Generic (PLEG): container finished" podID="a1583ba0-2c91-4348-bfd0-83e2b43bac2c" containerID="0c58fede203767ba8bc9bdd8f109150df795dc63f1482605d1b9eb99911007a6" exitCode=0
Oct 30 19:52:10.874318 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.873907    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst" event={"ID":"a1583ba0-2c91-4348-bfd0-83e2b43bac2c","Type":"ContainerDied","Data":"0c58fede203767ba8bc9bdd8f109150df795dc63f1482605d1b9eb99911007a6"}
Oct 30 19:52:10.875663 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.875641    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"
Oct 30 19:52:10.875663 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.875650    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l" event={"ID":"cc739f79-1809-42c5-8660-816cfa106a70","Type":"ContainerDied","Data":"15b16276dabf05c317c585bbc89b20ab57e7d51c84ba66870b187ee6c79ec15c"}
Oct 30 19:52:10.875923 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.875691    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15b16276dabf05c317c585bbc89b20ab57e7d51c84ba66870b187ee6c79ec15c"
Oct 30 19:52:10.877927 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.877903    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh" event={"ID":"76698bf7-4947-4e62-8bda-b4e2a17c44f5","Type":"ContainerDied","Data":"5abffb40e1a18a780ee454627b4e202869b8e3eea3c68cdaad32206159f6412c"}
Oct 30 19:52:10.878040 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.877946    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5abffb40e1a18a780ee454627b4e202869b8e3eea3c68cdaad32206159f6412c"
Oct 30 19:52:10.878040 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:10.878000    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"
Oct 30 19:52:12.055143 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:12.055116    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"
Oct 30 19:52:12.170486 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:12.170400    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-maas-api-openshift-ca\") pod \"a1583ba0-2c91-4348-bfd0-83e2b43bac2c\" (UID: \"a1583ba0-2c91-4348-bfd0-83e2b43bac2c\") "
Oct 30 19:52:12.170654 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:12.170547    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-kube-api-access-k5n55\" (UniqueName: \"kubernetes.io/projected/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-kube-api-access-k5n55\") pod \"a1583ba0-2c91-4348-bfd0-83e2b43bac2c\" (UID: \"a1583ba0-2c91-4348-bfd0-83e2b43bac2c\") "
Oct 30 19:52:12.170654 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:12.170619    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-maas-api-xks-ca\") pod \"a1583ba0-2c91-4348-bfd0-83e2b43bac2c\" (UID: \"a1583ba0-2c91-4348-bfd0-83e2b43bac2c\") "
Oct 30 19:52:12.170837 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:12.170802    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-maas-api-openshift-ca" pod "a1583ba0-2c91-4348-bfd0-83e2b43bac2c" (UID: "a1583ba0-2c91-4348-bfd0-83e2b43bac2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:12.170986 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:12.170954    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:12.172699 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:12.172668    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-maas-api-xks-ca" pod "a1583ba0-2c91-4348-bfd0-83e2b43bac2c" (UID: "a1583ba0-2c91-4348-bfd0-83e2b43bac2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:12.172793 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:12.172676    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-kube-api-access-k5n55" pod "a1583ba0-2c91-4348-bfd0-83e2b43bac2c" (UID: "a1583ba0-2c91-4348-bfd0-83e2b43bac2c"). InnerVolumeSpecName "kube-api-access-k5n55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:12.272125 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:12.272094    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5n55\" (UniqueName: \"kubernetes.io/projected/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-kube-api-access-k5n55\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:12.272125 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:12.272126    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1583ba0-2c91-4348-bfd0-83e2b43bac2c-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:12.892456 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:12.892420    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"
Oct 30 19:52:12.892623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:12.892452    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst" event={"ID":"a1583ba0-2c91-4348-bfd0-83e2b43bac2c","Type":"ContainerDied","Data":"ac928aed694c55e103bf16a98b24e431728e6608f7c63c23abc84cd08a3fb95b"}
Oct 30 19:52:12.892623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:12.892504    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac928aed694c55e103bf16a98b24e431728e6608f7c63c23abc84cd08a3fb95b"
Oct 30 19:52:17.117894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.115776    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"]
Oct 30 19:52:17.138881 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.127887    4823 cpu_manager.go:414] "containerMap: removing container" podUID="cc739f79-1809-42c5-8660-816cfa106a70" containerName="revoke-keys"
Oct 30 19:52:17.138881 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.127935    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc739f79-1809-42c5-8660-816cfa106a70" containerName="revoke-keys"
Oct 30 19:52:17.138881 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.128023    4823 cpu_manager.go:414] "containerMap: removing container" podUID="a1583ba0-2c91-4348-bfd0-83e2b43bac2c" containerName="revoke-keys"
Oct 30 19:52:17.138881 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.128034    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1583ba0-2c91-4348-bfd0-83e2b43bac2c" containerName="revoke-keys"
Oct 30 19:52:17.138881 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.128063    4823 cpu_manager.go:414] "containerMap: removing container" podUID="76698bf7-4947-4e62-8bda-b4e2a17c44f5" containerName="revoke-keys"
Oct 30 19:52:17.138881 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.128069    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76698bf7-4947-4e62-8bda-b4e2a17c44f5" containerName="revoke-keys"
Oct 30 19:52:17.138881 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.128267    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="76698bf7-4947-4e62-8bda-b4e2a17c44f5" containerName="revoke-keys"
Oct 30 19:52:17.138881 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.128284    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc739f79-1809-42c5-8660-816cfa106a70" containerName="revoke-keys"
Oct 30 19:52:17.138881 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.128303    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1583ba0-2c91-4348-bfd0-83e2b43bac2c" containerName="revoke-keys"
Oct 30 19:52:17.231844 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.210664    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"]
Oct 30 19:52:17.231844 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.210784    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.247891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.242248    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ced832\"/\"distinct-2-w0-ced832-kserve-self-signed-certs\""
Oct 30 19:52:17.342888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.320905    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-tmp-dir\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.342888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.341141    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-home\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.342888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.341200    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mncgt\" (UniqueName: \"kubernetes.io/projected/795d618e-c7a6-4495-8784-9e5c8e1b00f0-kube-api-access-mncgt\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.342888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.341321    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-model-cache\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.342888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.341361    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-dshm\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.342888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.341385    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/795d618e-c7a6-4495-8784-9e5c8e1b00f0-tls-certs\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.486025 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.485928    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-tmp-dir\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.486766 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.486741    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-home\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.487253 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.487233    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mncgt\" (UniqueName: \"kubernetes.io/projected/795d618e-c7a6-4495-8784-9e5c8e1b00f0-kube-api-access-mncgt\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.487729 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.487709    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-model-cache\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.488208 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.488190    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-dshm\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.488679 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.488660    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/795d618e-c7a6-4495-8784-9e5c8e1b00f0-tls-certs\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.489393 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.488129    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-model-cache\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.489535 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.486596    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-tmp-dir\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.489643 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.487183    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-home\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.509499 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.509438    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mncgt\" (UniqueName: \"kubernetes.io/projected/795d618e-c7a6-4495-8784-9e5c8e1b00f0-kube-api-access-mncgt\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.568965 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.568913    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"]
Oct 30 19:52:17.581450 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.581420    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"
Oct 30 19:52:17.591818 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.589665    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:52:17.591818 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.590417    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:52:17.591818 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.590526    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:52:17.591818 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.590575    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:52:17.591818 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.591506    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddpqp\" (UniqueName: \"kubernetes.io/projected/af718ff0-4c4e-43a3-b801-6adf02ea1223-kube-api-access-ddpqp\") pod \"maas-api-revoke-sub-304ef3c6-ghclh\" (UID: \"af718ff0-4c4e-43a3-b801-6adf02ea1223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"
Oct 30 19:52:17.591818 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.591548    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af718ff0-4c4e-43a3-b801-6adf02ea1223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-304ef3c6-ghclh\" (UID: \"af718ff0-4c4e-43a3-b801-6adf02ea1223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"
Oct 30 19:52:17.591818 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.591660    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af718ff0-4c4e-43a3-b801-6adf02ea1223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-304ef3c6-ghclh\" (UID: \"af718ff0-4c4e-43a3-b801-6adf02ea1223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"
Oct 30 19:52:17.632326 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.632201    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"]
Oct 30 19:52:17.632326 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.632262    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/795d618e-c7a6-4495-8784-9e5c8e1b00f0-tls-certs\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.685841 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.685797    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-dshm\") pod \"distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") " pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.693166 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.692496    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddpqp\" (UniqueName: \"kubernetes.io/projected/af718ff0-4c4e-43a3-b801-6adf02ea1223-kube-api-access-ddpqp\") pod \"maas-api-revoke-sub-304ef3c6-ghclh\" (UID: \"af718ff0-4c4e-43a3-b801-6adf02ea1223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"
Oct 30 19:52:17.693166 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.692545    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af718ff0-4c4e-43a3-b801-6adf02ea1223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-304ef3c6-ghclh\" (UID: \"af718ff0-4c4e-43a3-b801-6adf02ea1223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"
Oct 30 19:52:17.693166 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.692639    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af718ff0-4c4e-43a3-b801-6adf02ea1223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-304ef3c6-ghclh\" (UID: \"af718ff0-4c4e-43a3-b801-6adf02ea1223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"
Oct 30 19:52:17.715910 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.695809    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af718ff0-4c4e-43a3-b801-6adf02ea1223-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-304ef3c6-ghclh\" (UID: \"af718ff0-4c4e-43a3-b801-6adf02ea1223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"
Oct 30 19:52:17.729894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.720369    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af718ff0-4c4e-43a3-b801-6adf02ea1223-maas-api-xks-ca\") pod \"maas-api-revoke-sub-304ef3c6-ghclh\" (UID: \"af718ff0-4c4e-43a3-b801-6adf02ea1223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"
Oct 30 19:52:17.765248 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.757595    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddpqp\" (UniqueName: \"kubernetes.io/projected/af718ff0-4c4e-43a3-b801-6adf02ea1223-kube-api-access-ddpqp\") pod \"maas-api-revoke-sub-304ef3c6-ghclh\" (UID: \"af718ff0-4c4e-43a3-b801-6adf02ea1223\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"
Oct 30 19:52:17.843884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.843135    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:17.998904 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:17.976096    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"
Oct 30 19:52:18.923899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:18.921956    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:19.091576 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:19.091469    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:52:19.107369 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:19.092201    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:52:19.107688 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:19.092810    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:19.159297 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:19.159220    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:19.194959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:19.192722    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:19.260972 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:19.242424    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:19.480895 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:19.478121    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"]
Oct 30 19:52:19.781210 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:19.778545    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Oct 30 19:52:19.877399 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:19.868232    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"]
Oct 30 19:52:20.099029 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:20.095783    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:20.099029 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:20.096513    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:52:20.513911 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:20.505210    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:52:20.911498 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:20.911288    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:52:20.966123 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:20.965962    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:21.061932 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.059379    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:52:21.146383 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.146276    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:21.221951 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.187215    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"]
Oct 30 19:52:21.221951 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.196131    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:52:21.247204 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.245960    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"
Oct 30 19:52:21.247204 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.245851    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh" event={"ID":"af718ff0-4c4e-43a3-b801-6adf02ea1223","Type":"ContainerStarted","Data":"0757788e4009300277e89c22ed1dd0185ae3ad991bfb28b50a35eb3304cece61"}
Oct 30 19:52:21.247204 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.246716    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" event={"ID":"795d618e-c7a6-4495-8784-9e5c8e1b00f0","Type":"ContainerStarted","Data":"b99bc0bbe01816aa8b688452dbb5a8b74cf3aa8d5315052b825e2e4f07f5f625"}
Oct 30 19:52:21.247204 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.246740    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"]
Oct 30 19:52:21.247204 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.246799    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:52:21.277402 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.276930    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:52:21.393519 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.393398    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxhh9\" (UniqueName: \"kubernetes.io/projected/5e3b378b-2bd6-4895-87bd-e80f26d52c98-kube-api-access-nxhh9\") pod \"maas-api-revoke-sub-31b9b376-dvknj\" (UID: \"5e3b378b-2bd6-4895-87bd-e80f26d52c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"
Oct 30 19:52:21.393714 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.393577    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3b378b-2bd6-4895-87bd-e80f26d52c98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31b9b376-dvknj\" (UID: \"5e3b378b-2bd6-4895-87bd-e80f26d52c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"
Oct 30 19:52:21.393714 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.393606    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3b378b-2bd6-4895-87bd-e80f26d52c98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31b9b376-dvknj\" (UID: \"5e3b378b-2bd6-4895-87bd-e80f26d52c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"
Oct 30 19:52:21.496262 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.495400    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxhh9\" (UniqueName: \"kubernetes.io/projected/5e3b378b-2bd6-4895-87bd-e80f26d52c98-kube-api-access-nxhh9\") pod \"maas-api-revoke-sub-31b9b376-dvknj\" (UID: \"5e3b378b-2bd6-4895-87bd-e80f26d52c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"
Oct 30 19:52:21.496262 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.495571    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3b378b-2bd6-4895-87bd-e80f26d52c98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31b9b376-dvknj\" (UID: \"5e3b378b-2bd6-4895-87bd-e80f26d52c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"
Oct 30 19:52:21.496262 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.495602    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3b378b-2bd6-4895-87bd-e80f26d52c98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31b9b376-dvknj\" (UID: \"5e3b378b-2bd6-4895-87bd-e80f26d52c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"
Oct 30 19:52:21.501172 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.497487    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3b378b-2bd6-4895-87bd-e80f26d52c98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31b9b376-dvknj\" (UID: \"5e3b378b-2bd6-4895-87bd-e80f26d52c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"
Oct 30 19:52:21.504404 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.504365    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3b378b-2bd6-4895-87bd-e80f26d52c98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31b9b376-dvknj\" (UID: \"5e3b378b-2bd6-4895-87bd-e80f26d52c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"
Oct 30 19:52:21.514983 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.514949    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxhh9\" (UniqueName: \"kubernetes.io/projected/5e3b378b-2bd6-4895-87bd-e80f26d52c98-kube-api-access-nxhh9\") pod \"maas-api-revoke-sub-31b9b376-dvknj\" (UID: \"5e3b378b-2bd6-4895-87bd-e80f26d52c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"
Oct 30 19:52:21.608989 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.608961    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"
Oct 30 19:52:21.911267 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:21.911233    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"]
Oct 30 19:52:21.922004 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:52:21.921973    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3b378b_2bd6_4895_87bd_e80f26d52c98.slice/crio-8f795044b745dfbb8b353ca43077542079aae0258425e791c2ed683b118a6a3f WatchSource:0}: Error finding container 8f795044b745dfbb8b353ca43077542079aae0258425e791c2ed683b118a6a3f: Status 404 returned error can't find the container with id 8f795044b745dfbb8b353ca43077542079aae0258425e791c2ed683b118a6a3f
Oct 30 19:52:22.236346 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:22.236317    4823 generic.go:356] "Generic (PLEG): container finished" podID="af718ff0-4c4e-43a3-b801-6adf02ea1223" containerID="7f1aece4f3ce712f60ad2a7f7b5f13bc6a0167e1c67e5553798c8ed875f3f1f1" exitCode=0
Oct 30 19:52:22.236740 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:22.236376    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh" event={"ID":"af718ff0-4c4e-43a3-b801-6adf02ea1223","Type":"ContainerDied","Data":"7f1aece4f3ce712f60ad2a7f7b5f13bc6a0167e1c67e5553798c8ed875f3f1f1"}
Oct 30 19:52:22.238043 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:22.238014    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj" event={"ID":"5e3b378b-2bd6-4895-87bd-e80f26d52c98","Type":"ContainerStarted","Data":"e924d470dbab6907de681b941b09576a4071b6e6ec8092dfddb9df0000089ad6"}
Oct 30 19:52:22.238180 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:22.238062    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj" event={"ID":"5e3b378b-2bd6-4895-87bd-e80f26d52c98","Type":"ContainerStarted","Data":"8f795044b745dfbb8b353ca43077542079aae0258425e791c2ed683b118a6a3f"}
Oct 30 19:52:22.239459 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:22.239436    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" event={"ID":"795d618e-c7a6-4495-8784-9e5c8e1b00f0","Type":"ContainerStarted","Data":"b0ebe5b25ec0ec2a4cd09f564bf2f443dadce7394d195e46df816835ff7154d0"}
Oct 30 19:52:22.292663 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:22.292617    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj" podStartSLOduration=2.292603155 podStartE2EDuration="2.292603155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:22.288922216 +0000 UTC m=+1342.181536540" watchObservedRunningTime="2026-10-30 19:52:22.292603155 +0000 UTC m=+1342.185217523"
Oct 30 19:52:22.312083 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:22.312041    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podStartSLOduration=5.312028851 podStartE2EDuration="5.312028851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:22.310587948 +0000 UTC m=+1342.203202273" watchObservedRunningTime="2026-10-30 19:52:22.312028851 +0000 UTC m=+1342.204643179"
Oct 30 19:52:22.844440 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:22.844407    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:23.248695 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:23.248658    4823 generic.go:356] "Generic (PLEG): container finished" podID="5e3b378b-2bd6-4895-87bd-e80f26d52c98" containerID="e924d470dbab6907de681b941b09576a4071b6e6ec8092dfddb9df0000089ad6" exitCode=0
Oct 30 19:52:23.249127 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:23.248723    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj" event={"ID":"5e3b378b-2bd6-4895-87bd-e80f26d52c98","Type":"ContainerDied","Data":"e924d470dbab6907de681b941b09576a4071b6e6ec8092dfddb9df0000089ad6"}
Oct 30 19:52:23.409746 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:23.409687    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"
Oct 30 19:52:23.523832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:23.523745    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af718ff0-4c4e-43a3-b801-6adf02ea1223-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af718ff0-4c4e-43a3-b801-6adf02ea1223-maas-api-openshift-ca\") pod \"af718ff0-4c4e-43a3-b801-6adf02ea1223\" (UID: \"af718ff0-4c4e-43a3-b801-6adf02ea1223\") "
Oct 30 19:52:23.524028 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:23.524005    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af718ff0-4c4e-43a3-b801-6adf02ea1223-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af718ff0-4c4e-43a3-b801-6adf02ea1223-maas-api-xks-ca\") pod \"af718ff0-4c4e-43a3-b801-6adf02ea1223\" (UID: \"af718ff0-4c4e-43a3-b801-6adf02ea1223\") "
Oct 30 19:52:23.524096 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:23.524080    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af718ff0-4c4e-43a3-b801-6adf02ea1223-kube-api-access-ddpqp\" (UniqueName: \"kubernetes.io/projected/af718ff0-4c4e-43a3-b801-6adf02ea1223-kube-api-access-ddpqp\") pod \"af718ff0-4c4e-43a3-b801-6adf02ea1223\" (UID: \"af718ff0-4c4e-43a3-b801-6adf02ea1223\") "
Oct 30 19:52:23.524187 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:23.524167    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af718ff0-4c4e-43a3-b801-6adf02ea1223-maas-api-openshift-ca" pod "af718ff0-4c4e-43a3-b801-6adf02ea1223" (UID: "af718ff0-4c4e-43a3-b801-6adf02ea1223"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:23.524506 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:23.524489    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af718ff0-4c4e-43a3-b801-6adf02ea1223-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:23.526154 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:23.526128    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af718ff0-4c4e-43a3-b801-6adf02ea1223-kube-api-access-ddpqp" pod "af718ff0-4c4e-43a3-b801-6adf02ea1223" (UID: "af718ff0-4c4e-43a3-b801-6adf02ea1223"). InnerVolumeSpecName "kube-api-access-ddpqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:23.526154 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:23.526139    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af718ff0-4c4e-43a3-b801-6adf02ea1223-maas-api-xks-ca" pod "af718ff0-4c4e-43a3-b801-6adf02ea1223" (UID: "af718ff0-4c4e-43a3-b801-6adf02ea1223"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:23.625769 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:23.625739    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af718ff0-4c4e-43a3-b801-6adf02ea1223-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:23.625769 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:23.625765    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddpqp\" (UniqueName: \"kubernetes.io/projected/af718ff0-4c4e-43a3-b801-6adf02ea1223-kube-api-access-ddpqp\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:24.257349 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:24.257324    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"
Oct 30 19:52:24.257775 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:24.257360    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh" event={"ID":"af718ff0-4c4e-43a3-b801-6adf02ea1223","Type":"ContainerDied","Data":"0757788e4009300277e89c22ed1dd0185ae3ad991bfb28b50a35eb3304cece61"}
Oct 30 19:52:24.257775 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:24.257410    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0757788e4009300277e89c22ed1dd0185ae3ad991bfb28b50a35eb3304cece61"
Oct 30 19:52:24.424998 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:24.424974    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"
Oct 30 19:52:24.536903 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:24.536789    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e3b378b-2bd6-4895-87bd-e80f26d52c98-kube-api-access-nxhh9\" (UniqueName: \"kubernetes.io/projected/5e3b378b-2bd6-4895-87bd-e80f26d52c98-kube-api-access-nxhh9\") pod \"5e3b378b-2bd6-4895-87bd-e80f26d52c98\" (UID: \"5e3b378b-2bd6-4895-87bd-e80f26d52c98\") "
Oct 30 19:52:24.537053 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:24.536976    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e3b378b-2bd6-4895-87bd-e80f26d52c98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3b378b-2bd6-4895-87bd-e80f26d52c98-maas-api-openshift-ca\") pod \"5e3b378b-2bd6-4895-87bd-e80f26d52c98\" (UID: \"5e3b378b-2bd6-4895-87bd-e80f26d52c98\") "
Oct 30 19:52:24.537053 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:24.537010    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e3b378b-2bd6-4895-87bd-e80f26d52c98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3b378b-2bd6-4895-87bd-e80f26d52c98-maas-api-xks-ca\") pod \"5e3b378b-2bd6-4895-87bd-e80f26d52c98\" (UID: \"5e3b378b-2bd6-4895-87bd-e80f26d52c98\") "
Oct 30 19:52:24.537341 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:24.537314    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e3b378b-2bd6-4895-87bd-e80f26d52c98-maas-api-openshift-ca" pod "5e3b378b-2bd6-4895-87bd-e80f26d52c98" (UID: "5e3b378b-2bd6-4895-87bd-e80f26d52c98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:24.537470 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:24.537432    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e3b378b-2bd6-4895-87bd-e80f26d52c98-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:24.539387 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:24.539360    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3b378b-2bd6-4895-87bd-e80f26d52c98-kube-api-access-nxhh9" pod "5e3b378b-2bd6-4895-87bd-e80f26d52c98" (UID: "5e3b378b-2bd6-4895-87bd-e80f26d52c98"). InnerVolumeSpecName "kube-api-access-nxhh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:24.539471 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:24.539453    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3b378b-2bd6-4895-87bd-e80f26d52c98-maas-api-xks-ca" pod "5e3b378b-2bd6-4895-87bd-e80f26d52c98" (UID: "5e3b378b-2bd6-4895-87bd-e80f26d52c98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:24.639031 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:24.638993    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxhh9\" (UniqueName: \"kubernetes.io/projected/5e3b378b-2bd6-4895-87bd-e80f26d52c98-kube-api-access-nxhh9\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:24.639031 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:24.639027    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e3b378b-2bd6-4895-87bd-e80f26d52c98-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:25.265737 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.265707    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"
Oct 30 19:52:25.266335 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.265701    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj" event={"ID":"5e3b378b-2bd6-4895-87bd-e80f26d52c98","Type":"ContainerDied","Data":"8f795044b745dfbb8b353ca43077542079aae0258425e791c2ed683b118a6a3f"}
Oct 30 19:52:25.266335 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.265850    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f795044b745dfbb8b353ca43077542079aae0258425e791c2ed683b118a6a3f"
Oct 30 19:52:25.410960 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.410927    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"]
Oct 30 19:52:25.411514 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.411497    4823 cpu_manager.go:414] "containerMap: removing container" podUID="5e3b378b-2bd6-4895-87bd-e80f26d52c98" containerName="revoke-keys"
Oct 30 19:52:25.411514 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.411514    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e3b378b-2bd6-4895-87bd-e80f26d52c98" containerName="revoke-keys"
Oct 30 19:52:25.411666 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.411555    4823 cpu_manager.go:414] "containerMap: removing container" podUID="af718ff0-4c4e-43a3-b801-6adf02ea1223" containerName="revoke-keys"
Oct 30 19:52:25.411666 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.411560    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af718ff0-4c4e-43a3-b801-6adf02ea1223" containerName="revoke-keys"
Oct 30 19:52:25.411666 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.411656    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e3b378b-2bd6-4895-87bd-e80f26d52c98" containerName="revoke-keys"
Oct 30 19:52:25.411822 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.411672    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="af718ff0-4c4e-43a3-b801-6adf02ea1223" containerName="revoke-keys"
Oct 30 19:52:25.415382 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.415360    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.447393 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.447360    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"]
Oct 30 19:52:25.550488 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.550394    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df66169-610e-4ff9-8706-25e02d655b48-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.550488 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.550456    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.550676 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.550541    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.550676 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.550566    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.550676 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.550588    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.550765 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.550718    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wd96\" (UniqueName: \"kubernetes.io/projected/6df66169-610e-4ff9-8706-25e02d655b48-kube-api-access-4wd96\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.550799 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.550761    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df66169-610e-4ff9-8706-25e02d655b48-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.550852 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.550832    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.550904 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.550872    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df66169-610e-4ff9-8706-25e02d655b48-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.652492 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.652457    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wd96\" (UniqueName: \"kubernetes.io/projected/6df66169-610e-4ff9-8706-25e02d655b48-kube-api-access-4wd96\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.652689 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.652503    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df66169-610e-4ff9-8706-25e02d655b48-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.652689 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.652544    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.652689 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.652565    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df66169-610e-4ff9-8706-25e02d655b48-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.652689 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.652672    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df66169-610e-4ff9-8706-25e02d655b48-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.652949 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.652710    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.652949 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.652745    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.652949 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.652774    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.652949 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.652806    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.653147 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.653110    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.653197 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.653166    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.653252 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.653227    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.653295 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.653269    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.653560 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.653532    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df66169-610e-4ff9-8706-25e02d655b48-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.655141 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.655121    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.655224 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.655158    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df66169-610e-4ff9-8706-25e02d655b48-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.695963 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.695928    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wd96\" (UniqueName: \"kubernetes.io/projected/6df66169-610e-4ff9-8706-25e02d655b48-kube-api-access-4wd96\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.696577 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.696548    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df66169-610e-4ff9-8706-25e02d655b48-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:25.730204 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:25.730171    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:26.029457 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:26.029426    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"]
Oct 30 19:52:26.030653 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:52:26.030624    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df66169_610e_4ff9_8706_25e02d655b48.slice/crio-f2a88e7bc057f07c3511d3ea8a0ff7154544eecbf00f4106fe1351018d119894 WatchSource:0}: Error finding container f2a88e7bc057f07c3511d3ea8a0ff7154544eecbf00f4106fe1351018d119894: Status 404 returned error can't find the container with id f2a88e7bc057f07c3511d3ea8a0ff7154544eecbf00f4106fe1351018d119894
Oct 30 19:52:26.033058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:26.033024    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:52:26.033146 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:26.033099    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:52:26.033202 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:26.033141    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:52:26.280389 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:26.280355    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p" event={"ID":"6df66169-610e-4ff9-8706-25e02d655b48","Type":"ContainerStarted","Data":"9cd9b1c5acdf48457bc4f6b5f47b8f9d7953971c9aad05516ef2e2a1720a46b0"}
Oct 30 19:52:26.280742 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:26.280409    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p" event={"ID":"6df66169-610e-4ff9-8706-25e02d655b48","Type":"ContainerStarted","Data":"f2a88e7bc057f07c3511d3ea8a0ff7154544eecbf00f4106fe1351018d119894"}
Oct 30 19:52:26.305814 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:26.305724    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p" podStartSLOduration=1.305710399 podStartE2EDuration="1.305710399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:26.305248068 +0000 UTC m=+1346.197862393" watchObservedRunningTime="2026-10-30 19:52:26.305710399 +0000 UTC m=+1346.198324722"
Oct 30 19:52:26.730716 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:26.730681    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:27.732385 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:27.732354    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:52:27.732761 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:27.732405    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p" podUID="6df66169-610e-4ff9-8706-25e02d655b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 19:52:27.844020 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:27.843987    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:27.883303 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:27.883272    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:28.302754 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:28.302728    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:28.730517 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:28.730487    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:52:28.730723 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:28.730555    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p" podUID="6df66169-610e-4ff9-8706-25e02d655b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 19:52:29.731534 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:29.731494    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:52:29.732057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:29.731558    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p" podUID="6df66169-610e-4ff9-8706-25e02d655b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 19:52:30.730661 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:30.730627    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:52:30.730838 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:30.730687    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p" podUID="6df66169-610e-4ff9-8706-25e02d655b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 19:52:31.731289 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:31.731252    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:52:31.731716 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:31.731319    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p" podUID="6df66169-610e-4ff9-8706-25e02d655b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 19:52:32.017649 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:52:32.017569    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:52:32.912618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:32.912584    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:52:33.042500 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:33.042465    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:33.043360 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:33.043329    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:33.043988 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:33.043968    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 19:52:36.675553 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:36.675515    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"]
Oct 30 19:52:36.681098 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:36.681077    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"
Oct 30 19:52:36.687794 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:36.687764    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:52:36.688341 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:36.687808    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:52:36.688341 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:36.687857    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:52:36.688341 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:36.688005    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:52:36.693244 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:36.693221    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"]
Oct 30 19:52:36.831799 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:36.831762    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ht79\" (UniqueName: \"kubernetes.io/projected/42128e23-e4a6-43d2-a22a-446c337ab727-kube-api-access-9ht79\") pod \"maas-api-revoke-sub-5a73316f-ksrbs\" (UID: \"42128e23-e4a6-43d2-a22a-446c337ab727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"
Oct 30 19:52:36.832025 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:36.831952    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42128e23-e4a6-43d2-a22a-446c337ab727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a73316f-ksrbs\" (UID: \"42128e23-e4a6-43d2-a22a-446c337ab727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"
Oct 30 19:52:36.832025 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:36.831993    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42128e23-e4a6-43d2-a22a-446c337ab727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a73316f-ksrbs\" (UID: \"42128e23-e4a6-43d2-a22a-446c337ab727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"
Oct 30 19:52:36.933228 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:36.933148    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ht79\" (UniqueName: \"kubernetes.io/projected/42128e23-e4a6-43d2-a22a-446c337ab727-kube-api-access-9ht79\") pod \"maas-api-revoke-sub-5a73316f-ksrbs\" (UID: \"42128e23-e4a6-43d2-a22a-446c337ab727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"
Oct 30 19:52:36.933382 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:36.933263    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42128e23-e4a6-43d2-a22a-446c337ab727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a73316f-ksrbs\" (UID: \"42128e23-e4a6-43d2-a22a-446c337ab727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"
Oct 30 19:52:36.933382 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:36.933291    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42128e23-e4a6-43d2-a22a-446c337ab727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a73316f-ksrbs\" (UID: \"42128e23-e4a6-43d2-a22a-446c337ab727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"
Oct 30 19:52:36.933959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:36.933937    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42128e23-e4a6-43d2-a22a-446c337ab727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a73316f-ksrbs\" (UID: \"42128e23-e4a6-43d2-a22a-446c337ab727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"
Oct 30 19:52:36.935680 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:36.935663    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42128e23-e4a6-43d2-a22a-446c337ab727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a73316f-ksrbs\" (UID: \"42128e23-e4a6-43d2-a22a-446c337ab727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"
Oct 30 19:52:36.945742 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:36.945720    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ht79\" (UniqueName: \"kubernetes.io/projected/42128e23-e4a6-43d2-a22a-446c337ab727-kube-api-access-9ht79\") pod \"maas-api-revoke-sub-5a73316f-ksrbs\" (UID: \"42128e23-e4a6-43d2-a22a-446c337ab727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"
Oct 30 19:52:37.000322 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:37.000291    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"
Oct 30 19:52:37.289687 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:37.289593    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"]
Oct 30 19:52:37.366211 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:37.366175    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs" event={"ID":"42128e23-e4a6-43d2-a22a-446c337ab727","Type":"ContainerStarted","Data":"971171fed2cb8539215119f99400d9e26508f44b9236464ef1e9e614e40160b1"}
Oct 30 19:52:38.374763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:38.374731    4823 generic.go:356] "Generic (PLEG): container finished" podID="42128e23-e4a6-43d2-a22a-446c337ab727" containerID="22cf4bf4655675e073e60393369d68b7e3886af022df2f5bf572fc8fae84e6ed" exitCode=0
Oct 30 19:52:38.375178 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:38.374787    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs" event={"ID":"42128e23-e4a6-43d2-a22a-446c337ab727","Type":"ContainerDied","Data":"22cf4bf4655675e073e60393369d68b7e3886af022df2f5bf572fc8fae84e6ed"}
Oct 30 19:52:39.567674 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:39.567640    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"
Oct 30 19:52:39.669296 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:39.669213    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42128e23-e4a6-43d2-a22a-446c337ab727-kube-api-access-9ht79\" (UniqueName: \"kubernetes.io/projected/42128e23-e4a6-43d2-a22a-446c337ab727-kube-api-access-9ht79\") pod \"42128e23-e4a6-43d2-a22a-446c337ab727\" (UID: \"42128e23-e4a6-43d2-a22a-446c337ab727\") "
Oct 30 19:52:39.669296 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:39.669258    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42128e23-e4a6-43d2-a22a-446c337ab727-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42128e23-e4a6-43d2-a22a-446c337ab727-maas-api-openshift-ca\") pod \"42128e23-e4a6-43d2-a22a-446c337ab727\" (UID: \"42128e23-e4a6-43d2-a22a-446c337ab727\") "
Oct 30 19:52:39.669498 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:39.669347    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42128e23-e4a6-43d2-a22a-446c337ab727-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42128e23-e4a6-43d2-a22a-446c337ab727-maas-api-xks-ca\") pod \"42128e23-e4a6-43d2-a22a-446c337ab727\" (UID: \"42128e23-e4a6-43d2-a22a-446c337ab727\") "
Oct 30 19:52:39.669639 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:39.669607    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42128e23-e4a6-43d2-a22a-446c337ab727-maas-api-openshift-ca" pod "42128e23-e4a6-43d2-a22a-446c337ab727" (UID: "42128e23-e4a6-43d2-a22a-446c337ab727"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:39.670026 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:39.670004    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42128e23-e4a6-43d2-a22a-446c337ab727-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:39.671339 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:39.671315    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42128e23-e4a6-43d2-a22a-446c337ab727-maas-api-xks-ca" pod "42128e23-e4a6-43d2-a22a-446c337ab727" (UID: "42128e23-e4a6-43d2-a22a-446c337ab727"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:39.671452 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:39.671427    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42128e23-e4a6-43d2-a22a-446c337ab727-kube-api-access-9ht79" pod "42128e23-e4a6-43d2-a22a-446c337ab727" (UID: "42128e23-e4a6-43d2-a22a-446c337ab727"). InnerVolumeSpecName "kube-api-access-9ht79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:39.771536 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:39.771503    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ht79\" (UniqueName: \"kubernetes.io/projected/42128e23-e4a6-43d2-a22a-446c337ab727-kube-api-access-9ht79\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:39.771536 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:39.771532    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42128e23-e4a6-43d2-a22a-446c337ab727-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:52:39.981831 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:39.981755    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"]
Oct 30 19:52:39.982539 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:39.982513    4823 cpu_manager.go:414] "containerMap: removing container" podUID="42128e23-e4a6-43d2-a22a-446c337ab727" containerName="revoke-keys"
Oct 30 19:52:39.982539 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:39.982536    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42128e23-e4a6-43d2-a22a-446c337ab727" containerName="revoke-keys"
Oct 30 19:52:39.982740 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:39.982662    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="42128e23-e4a6-43d2-a22a-446c337ab727" containerName="revoke-keys"
Oct 30 19:52:39.986613 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:39.986593    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.005044 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.005017    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"]
Oct 30 19:52:40.075533 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.075498    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ad80bfe-3826-4dd0-bb18-c7710c865123-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.075533 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.075533    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2pbw\" (UniqueName: \"kubernetes.io/projected/0ad80bfe-3826-4dd0-bb18-c7710c865123-kube-api-access-h2pbw\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.075715 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.075556    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.075715 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.075690    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.075790 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.075732    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.075906 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.075892    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.075963 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.075917    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.076006 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.075991    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.076054 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.076040    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.126773 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.126749    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"]
Oct 30 19:52:40.130991 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.130973    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.151676 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.151652    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"]
Oct 30 19:52:40.177244 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.177211    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.177406 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.177342    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.177406 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.177372    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.177536 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.177421    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.177536 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.177450    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.177536 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.177500    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ad80bfe-3826-4dd0-bb18-c7710c865123-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.177536 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.177522    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2pbw\" (UniqueName: \"kubernetes.io/projected/0ad80bfe-3826-4dd0-bb18-c7710c865123-kube-api-access-h2pbw\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.177733 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.177548    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.177733 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.177591    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.178007 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.177981    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.178491 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.178465    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.178644 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.178613    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ad80bfe-3826-4dd0-bb18-c7710c865123-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.178724 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.178496    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.178795 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.178722    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.180964 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.180637    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.181247 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.181225    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.201469 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.201444    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2pbw\" (UniqueName: \"kubernetes.io/projected/0ad80bfe-3826-4dd0-bb18-c7710c865123-kube-api-access-h2pbw\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.201599 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.201475    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.279100 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.279010    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.279261 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.279121    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa894be2-b885-4001-912e-f6996634431f-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.279261 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.279170    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.279261 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.279188    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa894be2-b885-4001-912e-f6996634431f-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.279450 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.279274    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.279450 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.279339    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.279450 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.279364    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.279450 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.279399    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sn89\" (UniqueName: \"kubernetes.io/projected/fa894be2-b885-4001-912e-f6996634431f-kube-api-access-6sn89\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.279660 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.279450    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa894be2-b885-4001-912e-f6996634431f-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.305818 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.305789    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:40.391899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.382994    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.391899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.383043    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa894be2-b885-4001-912e-f6996634431f-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.391899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.383105    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.391899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.383215    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.391899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.383609    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.391899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.383913    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.391899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.383965    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.391899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.384027    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sn89\" (UniqueName: \"kubernetes.io/projected/fa894be2-b885-4001-912e-f6996634431f-kube-api-access-6sn89\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.391899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.384067    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa894be2-b885-4001-912e-f6996634431f-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.391899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.384143    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.391899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.384962    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa894be2-b885-4001-912e-f6996634431f-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.391899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.388614    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.391899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.389273    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa894be2-b885-4001-912e-f6996634431f-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.391899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.389450    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.405311 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.400228    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.405311 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.403351    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa894be2-b885-4001-912e-f6996634431f-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.405311 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.403830    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa894be2-b885-4001-912e-f6996634431f-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.410411 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.410331    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sn89\" (UniqueName: \"kubernetes.io/projected/fa894be2-b885-4001-912e-f6996634431f-kube-api-access-6sn89\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.414148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.413119    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs" event={"ID":"42128e23-e4a6-43d2-a22a-446c337ab727","Type":"ContainerDied","Data":"971171fed2cb8539215119f99400d9e26508f44b9236464ef1e9e614e40160b1"}
Oct 30 19:52:40.414148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.413185    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="971171fed2cb8539215119f99400d9e26508f44b9236464ef1e9e614e40160b1"
Oct 30 19:52:40.414148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.413272    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"
Oct 30 19:52:40.444939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.444900    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:40.613370 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.613325    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"]
Oct 30 19:52:40.613802 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:52:40.613622    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad80bfe_3826_4dd0_bb18_c7710c865123.slice/crio-0e6e670f3855266428efe1743f213f16b1bfcc502a38717a405cd6541e94c479 WatchSource:0}: Error finding container 0e6e670f3855266428efe1743f213f16b1bfcc502a38717a405cd6541e94c479: Status 404 returned error can't find the container with id 0e6e670f3855266428efe1743f213f16b1bfcc502a38717a405cd6541e94c479
Oct 30 19:52:40.616607 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.616570    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:52:40.616681 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.616656    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:52:40.616730 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.616719    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:52:40.778786 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.778754    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"]
Oct 30 19:52:40.779388 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:52:40.779360    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa894be2_b885_4001_912e_f6996634431f.slice/crio-852dfc64046f10bb3738a0a5e3277e57f8a44b3d68b189e56cc5849107d1bc53 WatchSource:0}: Error finding container 852dfc64046f10bb3738a0a5e3277e57f8a44b3d68b189e56cc5849107d1bc53: Status 404 returned error can't find the container with id 852dfc64046f10bb3738a0a5e3277e57f8a44b3d68b189e56cc5849107d1bc53
Oct 30 19:52:40.782467 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.782434    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:52:40.782577 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.782511    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:52:40.782577 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.782557    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:52:40.885482 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.885298    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"]
Oct 30 19:52:40.899166 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.899137    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:40.915218 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.915191    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"]
Oct 30 19:52:40.988073 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.988041    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"]
Oct 30 19:52:40.992291 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.992269    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:40.992613 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.992591    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:40.992700 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.992622    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762ae046-3cc6-4089-891d-5e4fb76ad039-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:40.992760 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.992697    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqslp\" (UniqueName: \"kubernetes.io/projected/762ae046-3cc6-4089-891d-5e4fb76ad039-kube-api-access-qqslp\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:40.992817 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.992802    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:40.992898 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.992827    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:40.992898 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.992851    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:40.993010 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.992919    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:40.993137 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.993123    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:40.993209 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:40.993156    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.024612 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.024579    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"]
Oct 30 19:52:41.094066 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.093975    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.094066 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.094009    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.094066 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.094039    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.094352 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.094090    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.094352 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.094129    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.094352 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.094244    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7954659-6a11-45ba-80e1-af0684ae90a3-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.094352 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.094272    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwc2p\" (UniqueName: \"kubernetes.io/projected/f7954659-6a11-45ba-80e1-af0684ae90a3-kube-api-access-wwc2p\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.094352 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.094299    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.094619 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.094576    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.094619 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.094611    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.094724 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.094645    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.094724 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.094667    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.094823 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.094790    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.094920 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.094835    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.094992 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.094948    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.094992 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.094974    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762ae046-3cc6-4089-891d-5e4fb76ad039-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.095093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.094998    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqslp\" (UniqueName: \"kubernetes.io/projected/762ae046-3cc6-4089-891d-5e4fb76ad039-kube-api-access-qqslp\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.095093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.095010    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.095093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.095029    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.095093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.095060    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.095396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.095374    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.095825 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.095798    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762ae046-3cc6-4089-891d-5e4fb76ad039-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.095970 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.095955    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.096843 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.096824    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.097821 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.097798    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.123298 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.123265    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.124500 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.124478    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqslp\" (UniqueName: \"kubernetes.io/projected/762ae046-3cc6-4089-891d-5e4fb76ad039-kube-api-access-qqslp\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.198487 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.198453    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.198487 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.198494    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.198728 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.198522    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7954659-6a11-45ba-80e1-af0684ae90a3-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.198728 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.198543    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwc2p\" (UniqueName: \"kubernetes.io/projected/f7954659-6a11-45ba-80e1-af0684ae90a3-kube-api-access-wwc2p\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.198728 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.198671    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.198728 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.198698    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.198959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.198736    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.198959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.198854    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.198959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.198935    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.199516 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.199490    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.199842 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.199804    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.203675 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.203649    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.204356 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.204324    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.204461 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.204368    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7954659-6a11-45ba-80e1-af0684ae90a3-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.207543 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.207509    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.209203 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.209179    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.216249 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.216221    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.217104 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.217080    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwc2p\" (UniqueName: \"kubernetes.io/projected/f7954659-6a11-45ba-80e1-af0684ae90a3-kube-api-access-wwc2p\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.232315 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.232206    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:41.309683 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.308221    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:41.435904 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.435683    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz" event={"ID":"0ad80bfe-3826-4dd0-bb18-c7710c865123","Type":"ContainerStarted","Data":"dc8b7d1f9784a99c668afa8f1a2a864de50fe773d927728b4b826bfcb3e4a86d"}
Oct 30 19:52:41.436575 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.436366    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz" event={"ID":"0ad80bfe-3826-4dd0-bb18-c7710c865123","Type":"ContainerStarted","Data":"0e6e670f3855266428efe1743f213f16b1bfcc502a38717a405cd6541e94c479"}
Oct 30 19:52:41.439592 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.439566    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl" event={"ID":"fa894be2-b885-4001-912e-f6996634431f","Type":"ContainerStarted","Data":"bcfcf3a305c97bba0cd57e7e390c2b238567407929c693487f13a1faa5d78869"}
Oct 30 19:52:41.439795 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.439775    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl" event={"ID":"fa894be2-b885-4001-912e-f6996634431f","Type":"ContainerStarted","Data":"852dfc64046f10bb3738a0a5e3277e57f8a44b3d68b189e56cc5849107d1bc53"}
Oct 30 19:52:41.446769 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.446743    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:41.446986 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.446964    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:52:41.447069 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.447009    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl" podUID="fa894be2-b885-4001-912e-f6996634431f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 19:52:41.488734 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.488674    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz" podStartSLOduration=2.488655315 podStartE2EDuration="2.488655315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:41.46536724 +0000 UTC m=+1361.357981565" watchObservedRunningTime="2026-10-30 19:52:41.488655315 +0000 UTC m=+1361.381269639"
Oct 30 19:52:41.521272 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.521207    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl" podStartSLOduration=1.5211865580000001 podStartE2EDuration="1.521186558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:41.511180018 +0000 UTC m=+1361.403794343" watchObservedRunningTime="2026-10-30 19:52:41.521186558 +0000 UTC m=+1361.413800884"
Oct 30 19:52:41.587436 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:52:41.587407    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762ae046_3cc6_4089_891d_5e4fb76ad039.slice/crio-b8f8ee27da5998c88642288dd80972a9a78f5e3542ddd4c22eb58c8eb21052cf WatchSource:0}: Error finding container b8f8ee27da5998c88642288dd80972a9a78f5e3542ddd4c22eb58c8eb21052cf: Status 404 returned error can't find the container with id b8f8ee27da5998c88642288dd80972a9a78f5e3542ddd4c22eb58c8eb21052cf
Oct 30 19:52:41.590849 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.590810    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:52:41.590992 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.590929    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:52:41.590992 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.590983    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:52:41.591061 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.591007    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"]
Oct 30 19:52:41.634807 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.634768    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"]
Oct 30 19:52:41.645226 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:52:41.645169    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7954659_6a11_45ba_80e1_af0684ae90a3.slice/crio-4ac965aa3c67d73f22c6010947ac2996109dd74fb835bc511784d08f5434be3c WatchSource:0}: Error finding container 4ac965aa3c67d73f22c6010947ac2996109dd74fb835bc511784d08f5434be3c: Status 404 returned error can't find the container with id 4ac965aa3c67d73f22c6010947ac2996109dd74fb835bc511784d08f5434be3c
Oct 30 19:52:41.648696 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.648665    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:52:41.648797 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.648731    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:52:41.648797 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:41.648760    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:52:42.316168 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:42.315836    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:52:42.316168 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:42.315911    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz" podUID="0ad80bfe-3826-4dd0-bb18-c7710c865123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:52:42.324509 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:42.324484    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:42.445318 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:42.445285    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:52:42.445464 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:42.445333    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl" podUID="fa894be2-b885-4001-912e-f6996634431f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 19:52:42.448889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:42.448833    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj" event={"ID":"f7954659-6a11-45ba-80e1-af0684ae90a3","Type":"ContainerStarted","Data":"86722aa378879ef0acba47fea75d9ad0214951be34d48030850f5e42449b71aa"}
Oct 30 19:52:42.449015 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:42.448912    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj" event={"ID":"f7954659-6a11-45ba-80e1-af0684ae90a3","Type":"ContainerStarted","Data":"4ac965aa3c67d73f22c6010947ac2996109dd74fb835bc511784d08f5434be3c"}
Oct 30 19:52:42.450571 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:42.450543    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq" event={"ID":"762ae046-3cc6-4089-891d-5e4fb76ad039","Type":"ContainerStarted","Data":"b2b1a4f9a5a8e1d82a65e407a495bcd33dd6c600ee75cc5d08fa6439b0a1a889"}
Oct 30 19:52:42.450770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:42.450753    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq" event={"ID":"762ae046-3cc6-4089-891d-5e4fb76ad039","Type":"ContainerStarted","Data":"b8f8ee27da5998c88642288dd80972a9a78f5e3542ddd4c22eb58c8eb21052cf"}
Oct 30 19:52:42.515901 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:42.485820    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj" podStartSLOduration=2.485800807 podStartE2EDuration="2.485800807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:42.478437384 +0000 UTC m=+1362.371051716" watchObservedRunningTime="2026-10-30 19:52:42.485800807 +0000 UTC m=+1362.378415134"
Oct 30 19:52:42.528936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:42.517398    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq" podStartSLOduration=2.5173759860000002 podStartE2EDuration="2.517375986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:42.51723752 +0000 UTC m=+1362.409851905" watchObservedRunningTime="2026-10-30 19:52:42.517375986 +0000 UTC m=+1362.409990312"
Oct 30 19:52:43.249886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:43.235724    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:43.249886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:43.235875    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:52:43.249886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:43.235925    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq" podUID="762ae046-3cc6-4089-891d-5e4fb76ad039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:52:43.322929 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:43.321930    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:43.355886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:43.338000    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:52:43.355886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:43.338064    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz" podUID="0ad80bfe-3826-4dd0-bb18-c7710c865123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:52:43.355886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:43.351604    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:52:43.355886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:43.351676    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj" podUID="f7954659-6a11-45ba-80e1-af0684ae90a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 19:52:43.456884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:43.456723    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:52:43.456884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:43.456785    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl" podUID="fa894be2-b885-4001-912e-f6996634431f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 19:52:43.952027 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:43.951979    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:44.110999 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.110950    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:44.202371 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.202246    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:44.254911 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.239833    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:52:44.254911 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.239913    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq" podUID="762ae046-3cc6-4089-891d-5e4fb76ad039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:52:44.310899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.306456    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:52:44.310899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.306514    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz" podUID="0ad80bfe-3826-4dd0-bb18-c7710c865123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:52:44.310899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.309243    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:52:44.310899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.309284    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj" podUID="f7954659-6a11-45ba-80e1-af0684ae90a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 19:52:44.341551 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.331049    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:52:44.341551 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.332249    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:52:44.455947 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.449229    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:52:44.455947 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.449289    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl" podUID="fa894be2-b885-4001-912e-f6996634431f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 19:52:44.580895 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.577025    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:44.946898 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.934731    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:44.965938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.950944    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:44.965938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.953592    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:44.965938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:44.953939    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:52:45.244947 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:45.220601    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:45.274914 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:45.242027    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:45.275977 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:45.264851    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:52:45.276229 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:45.276185    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq" podUID="762ae046-3cc6-4089-891d-5e4fb76ad039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:52:45.276421 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:45.273979    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:52:45.408903 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:45.372747    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:52:45.408903 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:45.373956    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:45.408903 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:45.384132    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:45.408903 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:45.390512    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:52:45.408903 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:45.390586    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz" podUID="0ad80bfe-3826-4dd0-bb18-c7710c865123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:52:45.437888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:45.417399    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:52:45.437888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:45.419198    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:52:45.437888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:45.419364    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj" podUID="f7954659-6a11-45ba-80e1-af0684ae90a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 19:52:45.966548 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:45.966490    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:46.170885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.090988    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:52:46.302357 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.302260    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:46.623518 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.203576    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:46.623819 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.201907    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:52:46.645548 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.344605    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:52:46.645852 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.645812    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl" podUID="fa894be2-b885-4001-912e-f6996634431f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 19:52:46.659243 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.407782    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:46.660596 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.596947    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:52:46.661726 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.406291    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:46.662402 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.386713    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:52:46.662560 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.662535    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj" podUID="f7954659-6a11-45ba-80e1-af0684ae90a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 19:52:46.662677 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.504140    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:52:46.662780 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.662764    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq" podUID="762ae046-3cc6-4089-891d-5e4fb76ad039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:52:46.662906 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.588528    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:52:46.663040 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.663018    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz" podUID="0ad80bfe-3826-4dd0-bb18-c7710c865123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:52:46.668732 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.668706    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:52:46.668951 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.668926    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl" podUID="fa894be2-b885-4001-912e-f6996634431f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 19:52:46.720895 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.705421    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:52:46.967542 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:46.967424    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:47.175189 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.174980    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:52:47.260982 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.259916    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:52:47.289959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.289915    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:52:47.290236 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.290205    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq" podUID="762ae046-3cc6-4089-891d-5e4fb76ad039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:52:47.329477 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.329437    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:52:47.370512 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.370108    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz" podUID="0ad80bfe-3826-4dd0-bb18-c7710c865123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:52:47.370920 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.338144    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:47.387899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.340240    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:52:47.387899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.387008    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj" podUID="f7954659-6a11-45ba-80e1-af0684ae90a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 19:52:47.511961 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.500194    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:52:47.511961 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.500257    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl" podUID="fa894be2-b885-4001-912e-f6996634431f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 19:52:47.665933 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.656959    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:52:47.826944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.748520    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:52:47.826944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.749227    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:47.826944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.751214    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:47.826944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.826461    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:52:47.895951 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:47.895913    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:48.177749 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.177701    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:48.194132 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.194092    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 19:52:48.249343 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.249275    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:52:48.249705 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.249669    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq" podUID="762ae046-3cc6-4089-891d-5e4fb76ad039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:52:48.334475 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.334438    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:52:48.335145 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.335051    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz" podUID="0ad80bfe-3826-4dd0-bb18-c7710c865123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:52:48.335397 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.335377    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:52:48.335548 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.335527    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj" podUID="f7954659-6a11-45ba-80e1-af0684ae90a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 19:52:48.427119 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.427073    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:52:48.456711 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.456612    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:52:48.463266 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.457938    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl" podUID="fa894be2-b885-4001-912e-f6996634431f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 19:52:48.601761 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.601731    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:52:48.682435 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.682254    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:48.775205 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.775096    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:48.794423 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.794383    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:52:48.829662 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.829620    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:48.871942 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.871882    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Oct 30 19:52:48.942587 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.942533    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:52:48.999202 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:48.999140    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:49.082897 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.082791    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:52:49.179390 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.178104    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:49.179390 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.178532    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:52:49.235396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.234988    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:52:49.235396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.235044    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq" podUID="762ae046-3cc6-4089-891d-5e4fb76ad039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:52:49.430037 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.429007    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:52:49.430037 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.429069    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz" podUID="0ad80bfe-3826-4dd0-bb18-c7710c865123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:52:49.430037 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.429161    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:49.430037 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.429654    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:52:49.430037 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.429692    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj" podUID="f7954659-6a11-45ba-80e1-af0684ae90a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 19:52:49.451044 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.451005    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:52:49.451206 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.451083    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl" podUID="fa894be2-b885-4001-912e-f6996634431f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 19:52:49.614031 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.613975    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:49.745938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.737880    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:52:49.745938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.737956    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:49.748879 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.746471    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:49.841886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.806948    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:49.936606 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:49.932273    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:52:50.031945 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:50.029415    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:52:50.032332 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:50.032304    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:52:50.033277 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:50.033179    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:52:50.044667 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:50.044646    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:52:50.052524 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:50.052505    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:52:50.222936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:50.205265    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:52:50.282103 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:50.282003    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:52:50.333536 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:50.333461    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:52:50.333827 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:50.333794    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq" podUID="762ae046-3cc6-4089-891d-5e4fb76ad039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:52:50.336255 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:50.336226    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:52:50.336422 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:50.336401    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj" podUID="f7954659-6a11-45ba-80e1-af0684ae90a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 19:52:50.413899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:50.413225    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:52:50.577276 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:50.577186    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:52:50.989850 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:50.989814    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:50.991437 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:50.991306    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:50.994512 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:50.994485    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 19:52:51.237157 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:51.237117    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:52:51.237359 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:51.237178    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq" podUID="762ae046-3cc6-4089-891d-5e4fb76ad039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:52:51.314071 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:51.310813    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:52:51.314071 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:51.310924    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj" podUID="f7954659-6a11-45ba-80e1-af0684ae90a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 19:52:51.365942 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:51.365829    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:51.366208 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:51.366177    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz" podUID="0ad80bfe-3826-4dd0-bb18-c7710c865123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:52.004882 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:51.999094    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:52.004882 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:52.000024    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:52.005893 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:52.005634    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 19:52:52.324630 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:52.324547    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:52.325095 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:52.325069    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:52.326073 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:52.326051    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 19:52:52.808488 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:52.808461    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:52.809003 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:52.808981    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:52.809856 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:52.809838    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 19:52:54.831202 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:52:54.831171    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:53:01.241620 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.241583    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"]
Oct 30 19:53:01.247600 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.247578    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"
Oct 30 19:53:01.261260 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.261227    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"]
Oct 30 19:53:01.309588 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.309559    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:53:01.309588 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.309583    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:53:01.309799 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.309763    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:53:01.309799 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.309765    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:53:01.373541 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.373506    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6579b770-b19c-47c3-8a5b-60d24481aec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d71580b-f8kwz\" (UID: \"6579b770-b19c-47c3-8a5b-60d24481aec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"
Oct 30 19:53:01.373728 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.373675    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6579b770-b19c-47c3-8a5b-60d24481aec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d71580b-f8kwz\" (UID: \"6579b770-b19c-47c3-8a5b-60d24481aec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"
Oct 30 19:53:01.373817 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.373797    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z88db\" (UniqueName: \"kubernetes.io/projected/6579b770-b19c-47c3-8a5b-60d24481aec9-kube-api-access-z88db\") pod \"maas-api-revoke-sub-3d71580b-f8kwz\" (UID: \"6579b770-b19c-47c3-8a5b-60d24481aec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"
Oct 30 19:53:01.475667 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.475629    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6579b770-b19c-47c3-8a5b-60d24481aec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d71580b-f8kwz\" (UID: \"6579b770-b19c-47c3-8a5b-60d24481aec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"
Oct 30 19:53:01.475928 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.475696    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z88db\" (UniqueName: \"kubernetes.io/projected/6579b770-b19c-47c3-8a5b-60d24481aec9-kube-api-access-z88db\") pod \"maas-api-revoke-sub-3d71580b-f8kwz\" (UID: \"6579b770-b19c-47c3-8a5b-60d24481aec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"
Oct 30 19:53:01.475928 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.475852    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6579b770-b19c-47c3-8a5b-60d24481aec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d71580b-f8kwz\" (UID: \"6579b770-b19c-47c3-8a5b-60d24481aec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"
Oct 30 19:53:01.476482 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.476454    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6579b770-b19c-47c3-8a5b-60d24481aec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d71580b-f8kwz\" (UID: \"6579b770-b19c-47c3-8a5b-60d24481aec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"
Oct 30 19:53:01.478282 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.478264    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6579b770-b19c-47c3-8a5b-60d24481aec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d71580b-f8kwz\" (UID: \"6579b770-b19c-47c3-8a5b-60d24481aec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"
Oct 30 19:53:01.490721 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.490689    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z88db\" (UniqueName: \"kubernetes.io/projected/6579b770-b19c-47c3-8a5b-60d24481aec9-kube-api-access-z88db\") pod \"maas-api-revoke-sub-3d71580b-f8kwz\" (UID: \"6579b770-b19c-47c3-8a5b-60d24481aec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"
Oct 30 19:53:01.565512 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.565438    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"
Oct 30 19:53:01.879970 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:01.879942    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"]
Oct 30 19:53:01.880632 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:53:01.880611    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6579b770_b19c_47c3_8a5b_60d24481aec9.slice/crio-20fd2932f0f90ce73360e42fbcae5f0a272c34b675cbd228ca6e292f43798ae1 WatchSource:0}: Error finding container 20fd2932f0f90ce73360e42fbcae5f0a272c34b675cbd228ca6e292f43798ae1: Status 404 returned error can't find the container with id 20fd2932f0f90ce73360e42fbcae5f0a272c34b675cbd228ca6e292f43798ae1
Oct 30 19:53:02.016998 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:53:02.016949    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:53:02.089555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:02.089520    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz" event={"ID":"6579b770-b19c-47c3-8a5b-60d24481aec9","Type":"ContainerStarted","Data":"20fd2932f0f90ce73360e42fbcae5f0a272c34b675cbd228ca6e292f43798ae1"}
Oct 30 19:53:03.099926 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:03.099814    4823 generic.go:356] "Generic (PLEG): container finished" podID="6579b770-b19c-47c3-8a5b-60d24481aec9" containerID="228a3b0ec23a290cda6c0e4bc34f7478f9c9ab81b969bad18a68fdb8f4d6250b" exitCode=0
Oct 30 19:53:03.099926 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:03.099905    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz" event={"ID":"6579b770-b19c-47c3-8a5b-60d24481aec9","Type":"ContainerDied","Data":"228a3b0ec23a290cda6c0e4bc34f7478f9c9ab81b969bad18a68fdb8f4d6250b"}
Oct 30 19:53:04.288163 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:04.288136    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"
Oct 30 19:53:04.426963 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:04.426878    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6579b770-b19c-47c3-8a5b-60d24481aec9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6579b770-b19c-47c3-8a5b-60d24481aec9-maas-api-xks-ca\") pod \"6579b770-b19c-47c3-8a5b-60d24481aec9\" (UID: \"6579b770-b19c-47c3-8a5b-60d24481aec9\") "
Oct 30 19:53:04.427157 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:04.427102    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6579b770-b19c-47c3-8a5b-60d24481aec9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6579b770-b19c-47c3-8a5b-60d24481aec9-maas-api-openshift-ca\") pod \"6579b770-b19c-47c3-8a5b-60d24481aec9\" (UID: \"6579b770-b19c-47c3-8a5b-60d24481aec9\") "
Oct 30 19:53:04.427157 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:04.427139    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6579b770-b19c-47c3-8a5b-60d24481aec9-kube-api-access-z88db\" (UniqueName: \"kubernetes.io/projected/6579b770-b19c-47c3-8a5b-60d24481aec9-kube-api-access-z88db\") pod \"6579b770-b19c-47c3-8a5b-60d24481aec9\" (UID: \"6579b770-b19c-47c3-8a5b-60d24481aec9\") "
Oct 30 19:53:04.427497 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:04.427470    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6579b770-b19c-47c3-8a5b-60d24481aec9-maas-api-openshift-ca" pod "6579b770-b19c-47c3-8a5b-60d24481aec9" (UID: "6579b770-b19c-47c3-8a5b-60d24481aec9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:04.429069 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:04.429039    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6579b770-b19c-47c3-8a5b-60d24481aec9-maas-api-xks-ca" pod "6579b770-b19c-47c3-8a5b-60d24481aec9" (UID: "6579b770-b19c-47c3-8a5b-60d24481aec9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:04.429213 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:04.429191    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6579b770-b19c-47c3-8a5b-60d24481aec9-kube-api-access-z88db" pod "6579b770-b19c-47c3-8a5b-60d24481aec9" (UID: "6579b770-b19c-47c3-8a5b-60d24481aec9"). InnerVolumeSpecName "kube-api-access-z88db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:04.528994 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:04.528955    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6579b770-b19c-47c3-8a5b-60d24481aec9-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:04.528994 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:04.528988    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6579b770-b19c-47c3-8a5b-60d24481aec9-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:04.529205 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:04.529003    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z88db\" (UniqueName: \"kubernetes.io/projected/6579b770-b19c-47c3-8a5b-60d24481aec9-kube-api-access-z88db\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:05.116260 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.116219    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"
Oct 30 19:53:05.116260 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.116237    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz" event={"ID":"6579b770-b19c-47c3-8a5b-60d24481aec9","Type":"ContainerDied","Data":"20fd2932f0f90ce73360e42fbcae5f0a272c34b675cbd228ca6e292f43798ae1"}
Oct 30 19:53:05.116550 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.116282    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20fd2932f0f90ce73360e42fbcae5f0a272c34b675cbd228ca6e292f43798ae1"
Oct 30 19:53:05.229024 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.228991    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"]
Oct 30 19:53:05.229683 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.229663    4823 cpu_manager.go:414] "containerMap: removing container" podUID="6579b770-b19c-47c3-8a5b-60d24481aec9" containerName="revoke-keys"
Oct 30 19:53:05.229801 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.229685    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6579b770-b19c-47c3-8a5b-60d24481aec9" containerName="revoke-keys"
Oct 30 19:53:05.229899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.229824    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="6579b770-b19c-47c3-8a5b-60d24481aec9" containerName="revoke-keys"
Oct 30 19:53:05.233613 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.233591    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.248855 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.248823    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"]
Oct 30 19:53:05.251791 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.251762    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-dshm\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.251944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.251815    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144ed52c-29ff-42f9-8695-701aef451ce5-tls-certs\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.251944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.251938    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-tmp-dir\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.252074 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.252035    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-model-cache\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.252162 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.252070    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ghtq\" (UniqueName: \"kubernetes.io/projected/144ed52c-29ff-42f9-8695-701aef451ce5-kube-api-access-7ghtq\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.252162 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.252144    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-home\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.257514 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.257487    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"distinct-2-w4-3f57c5-kserve-self-signed-certs\""
Oct 30 19:53:05.353082 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.353033    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-model-cache\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.353082 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.353089    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ghtq\" (UniqueName: \"kubernetes.io/projected/144ed52c-29ff-42f9-8695-701aef451ce5-kube-api-access-7ghtq\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.353673 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.353147    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-home\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.353673 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.353237    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-dshm\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.353673 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.353274    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144ed52c-29ff-42f9-8695-701aef451ce5-tls-certs\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.353673 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.353371    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-tmp-dir\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.353673 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.353485    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-model-cache\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.353673 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.353529    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-home\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.354014 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.353989    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-tmp-dir\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.355574 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.355553    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-dshm\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.355913 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.355891    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144ed52c-29ff-42f9-8695-701aef451ce5-tls-certs\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.365200 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.365173    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ghtq\" (UniqueName: \"kubernetes.io/projected/144ed52c-29ff-42f9-8695-701aef451ce5-kube-api-access-7ghtq\") pod \"distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.546045 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.545957    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:05.834321 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:53:05.834292    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod144ed52c_29ff_42f9_8695_701aef451ce5.slice/crio-9f17e44d1433908d7497f3dbbf453e4719b596cde0eea9cc2ce0f6869348dc2e WatchSource:0}: Error finding container 9f17e44d1433908d7497f3dbbf453e4719b596cde0eea9cc2ce0f6869348dc2e: Status 404 returned error can't find the container with id 9f17e44d1433908d7497f3dbbf453e4719b596cde0eea9cc2ce0f6869348dc2e
Oct 30 19:53:05.834840 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:05.834819    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"]
Oct 30 19:53:06.130519 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.130478    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" event={"ID":"144ed52c-29ff-42f9-8695-701aef451ce5","Type":"ContainerStarted","Data":"0db201eb8100ded6ef7992935a6e4c640a46ca6c230652e33e181277c19c03ab"}
Oct 30 19:53:06.130702 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.130533    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" event={"ID":"144ed52c-29ff-42f9-8695-701aef451ce5","Type":"ContainerStarted","Data":"9f17e44d1433908d7497f3dbbf453e4719b596cde0eea9cc2ce0f6869348dc2e"}
Oct 30 19:53:06.158683 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.158621    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podStartSLOduration=1.158602806 podStartE2EDuration="1.158602806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:06.151103816 +0000 UTC m=+1386.043718139" watchObservedRunningTime="2026-10-30 19:53:06.158602806 +0000 UTC m=+1386.051217131"
Oct 30 19:53:06.546458 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.546376    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:06.556919 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.556891    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"]
Oct 30 19:53:06.561598 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.561574    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.580231 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.580176    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 19:53:06.580376 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.580290    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 19:53:06.580376 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.580371    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 19:53:06.580623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.580609    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sh8sg\""
Oct 30 19:53:06.582068 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.582041    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"]
Oct 30 19:53:06.583284 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.583264    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.583375 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.583338    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eb246a-ee5d-4b22-8e01-f280a76b2521-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.583469 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.583447    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.583509 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.583485    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg2zk\" (UniqueName: \"kubernetes.io/projected/f8eb246a-ee5d-4b22-8e01-f280a76b2521-kube-api-access-dg2zk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.583540 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.583513    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.583610 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.583592    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.583720 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.583703    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.685074 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.685037    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eb246a-ee5d-4b22-8e01-f280a76b2521-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.685250 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.685099    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.685250 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.685127    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg2zk\" (UniqueName: \"kubernetes.io/projected/f8eb246a-ee5d-4b22-8e01-f280a76b2521-kube-api-access-dg2zk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.685341 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.685287    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.685382 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.685349    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.685506 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.685489    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.685598 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.685581    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.686354 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.686325    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.687495 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.687466    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eb246a-ee5d-4b22-8e01-f280a76b2521-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.687643 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.687621    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.687745 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.687728    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.687828 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.687810    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.704814 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.704782    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg2zk\" (UniqueName: \"kubernetes.io/projected/f8eb246a-ee5d-4b22-8e01-f280a76b2521-kube-api-access-dg2zk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.709038 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.709015    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l8kcm\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:06.880468 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:06.880440    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:07.184308 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:53:07.183695    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8eb246a_ee5d_4b22_8e01_f280a76b2521.slice/crio-4ca914f960ca27daf63b9a1a11201eaa1ae6745bd12429faf2e126a2571bc96b WatchSource:0}: Error finding container 4ca914f960ca27daf63b9a1a11201eaa1ae6745bd12429faf2e126a2571bc96b: Status 404 returned error can't find the container with id 4ca914f960ca27daf63b9a1a11201eaa1ae6745bd12429faf2e126a2571bc96b
Oct 30 19:53:07.184308 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:07.184039    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"]
Oct 30 19:53:08.342897 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:08.339824    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm" event={"ID":"f8eb246a-ee5d-4b22-8e01-f280a76b2521","Type":"ContainerStarted","Data":"4ca914f960ca27daf63b9a1a11201eaa1ae6745bd12429faf2e126a2571bc96b"}
Oct 30 19:53:08.666887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:08.666207    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:08.708887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:08.708223    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:09.216886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:09.214969    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:11.213149 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.213076    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"]
Oct 30 19:53:11.218581 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.218558    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"
Oct 30 19:53:11.228307 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.227911    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:53:11.228307 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.227961    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:53:11.228307 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.227997    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:53:11.228307 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.228074    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:53:11.233272 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.233249    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"]
Oct 30 19:53:11.276555 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.276509    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:11.276699 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.276597    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:11.317465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.317435    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqq9m\" (UniqueName: \"kubernetes.io/projected/b991fb45-8e0f-4fba-93e1-cf0086a825fe-kube-api-access-wqq9m\") pod \"maas-api-revoke-sub-37c2592d-7kbvf\" (UID: \"b991fb45-8e0f-4fba-93e1-cf0086a825fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"
Oct 30 19:53:11.317654 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.317633    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b991fb45-8e0f-4fba-93e1-cf0086a825fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37c2592d-7kbvf\" (UID: \"b991fb45-8e0f-4fba-93e1-cf0086a825fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"
Oct 30 19:53:11.317780 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.317765    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b991fb45-8e0f-4fba-93e1-cf0086a825fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37c2592d-7kbvf\" (UID: \"b991fb45-8e0f-4fba-93e1-cf0086a825fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"
Oct 30 19:53:11.373184 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.373147    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm" event={"ID":"f8eb246a-ee5d-4b22-8e01-f280a76b2521","Type":"ContainerStarted","Data":"45049a452b079902f910fdff9d69a24cc91d955c163c6314a88aaba34f730f5d"}
Oct 30 19:53:11.373342 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.373215    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:11.412354 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.412279    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm" podStartSLOduration=1.32390715 podStartE2EDuration="5.412259846s" totalImagesPullingTime="4.088352696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:11.410968847 +0000 UTC m=+1391.303583171" watchObservedRunningTime="2026-10-30 19:53:11.412259846 +0000 UTC m=+1391.304874172"
Oct 30 19:53:11.419071 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.419038    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b991fb45-8e0f-4fba-93e1-cf0086a825fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37c2592d-7kbvf\" (UID: \"b991fb45-8e0f-4fba-93e1-cf0086a825fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"
Oct 30 19:53:11.419285 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.419128    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqq9m\" (UniqueName: \"kubernetes.io/projected/b991fb45-8e0f-4fba-93e1-cf0086a825fe-kube-api-access-wqq9m\") pod \"maas-api-revoke-sub-37c2592d-7kbvf\" (UID: \"b991fb45-8e0f-4fba-93e1-cf0086a825fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"
Oct 30 19:53:11.419387 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.419370    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b991fb45-8e0f-4fba-93e1-cf0086a825fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37c2592d-7kbvf\" (UID: \"b991fb45-8e0f-4fba-93e1-cf0086a825fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"
Oct 30 19:53:11.420345 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.420310    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b991fb45-8e0f-4fba-93e1-cf0086a825fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37c2592d-7kbvf\" (UID: \"b991fb45-8e0f-4fba-93e1-cf0086a825fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"
Oct 30 19:53:11.421909 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.421884    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b991fb45-8e0f-4fba-93e1-cf0086a825fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37c2592d-7kbvf\" (UID: \"b991fb45-8e0f-4fba-93e1-cf0086a825fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"
Oct 30 19:53:11.435837 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.435810    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqq9m\" (UniqueName: \"kubernetes.io/projected/b991fb45-8e0f-4fba-93e1-cf0086a825fe-kube-api-access-wqq9m\") pod \"maas-api-revoke-sub-37c2592d-7kbvf\" (UID: \"b991fb45-8e0f-4fba-93e1-cf0086a825fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"
Oct 30 19:53:11.535425 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.535322    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"
Oct 30 19:53:11.862954 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:11.862925    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"]
Oct 30 19:53:11.863966 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:53:11.863923    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb991fb45_8e0f_4fba_93e1_cf0086a825fe.slice/crio-4bbb01b7c5318e500536765773461d1e65a431c2b1f2878ed89cc0ee9b906786 WatchSource:0}: Error finding container 4bbb01b7c5318e500536765773461d1e65a431c2b1f2878ed89cc0ee9b906786: Status 404 returned error can't find the container with id 4bbb01b7c5318e500536765773461d1e65a431c2b1f2878ed89cc0ee9b906786
Oct 30 19:53:12.382117 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:12.382081    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf" event={"ID":"b991fb45-8e0f-4fba-93e1-cf0086a825fe","Type":"ContainerStarted","Data":"ea3f1c47cb45bcb42771bd68e099d6a1da26d862708dafff295d261017b3697a"}
Oct 30 19:53:12.382548 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:12.382140    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf" event={"ID":"b991fb45-8e0f-4fba-93e1-cf0086a825fe","Type":"ContainerStarted","Data":"4bbb01b7c5318e500536765773461d1e65a431c2b1f2878ed89cc0ee9b906786"}
Oct 30 19:53:12.384190 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:12.384169    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 19:53:12.400985 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:12.400884    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf" podStartSLOduration=1.400855707 podStartE2EDuration="1.400855707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:12.400314545 +0000 UTC m=+1392.292928869" watchObservedRunningTime="2026-10-30 19:53:12.400855707 +0000 UTC m=+1392.293470034"
Oct 30 19:53:13.390849 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:13.390812    4823 generic.go:356] "Generic (PLEG): container finished" podID="b991fb45-8e0f-4fba-93e1-cf0086a825fe" containerID="ea3f1c47cb45bcb42771bd68e099d6a1da26d862708dafff295d261017b3697a" exitCode=0
Oct 30 19:53:13.391350 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:13.390903    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf" event={"ID":"b991fb45-8e0f-4fba-93e1-cf0086a825fe","Type":"ContainerDied","Data":"ea3f1c47cb45bcb42771bd68e099d6a1da26d862708dafff295d261017b3697a"}
Oct 30 19:53:14.563409 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:14.563381    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"
Oct 30 19:53:14.666970 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:14.666857    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b991fb45-8e0f-4fba-93e1-cf0086a825fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b991fb45-8e0f-4fba-93e1-cf0086a825fe-maas-api-openshift-ca\") pod \"b991fb45-8e0f-4fba-93e1-cf0086a825fe\" (UID: \"b991fb45-8e0f-4fba-93e1-cf0086a825fe\") "
Oct 30 19:53:14.667148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:14.666990    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b991fb45-8e0f-4fba-93e1-cf0086a825fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b991fb45-8e0f-4fba-93e1-cf0086a825fe-maas-api-xks-ca\") pod \"b991fb45-8e0f-4fba-93e1-cf0086a825fe\" (UID: \"b991fb45-8e0f-4fba-93e1-cf0086a825fe\") "
Oct 30 19:53:14.667191 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:14.667157    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b991fb45-8e0f-4fba-93e1-cf0086a825fe-kube-api-access-wqq9m\" (UniqueName: \"kubernetes.io/projected/b991fb45-8e0f-4fba-93e1-cf0086a825fe-kube-api-access-wqq9m\") pod \"b991fb45-8e0f-4fba-93e1-cf0086a825fe\" (UID: \"b991fb45-8e0f-4fba-93e1-cf0086a825fe\") "
Oct 30 19:53:14.667255 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:14.667232    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b991fb45-8e0f-4fba-93e1-cf0086a825fe-maas-api-openshift-ca" pod "b991fb45-8e0f-4fba-93e1-cf0086a825fe" (UID: "b991fb45-8e0f-4fba-93e1-cf0086a825fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:14.667679 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:14.667657    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b991fb45-8e0f-4fba-93e1-cf0086a825fe-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:14.669139 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:14.669109    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b991fb45-8e0f-4fba-93e1-cf0086a825fe-maas-api-xks-ca" pod "b991fb45-8e0f-4fba-93e1-cf0086a825fe" (UID: "b991fb45-8e0f-4fba-93e1-cf0086a825fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:14.669647 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:14.669616    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b991fb45-8e0f-4fba-93e1-cf0086a825fe-kube-api-access-wqq9m" pod "b991fb45-8e0f-4fba-93e1-cf0086a825fe" (UID: "b991fb45-8e0f-4fba-93e1-cf0086a825fe"). InnerVolumeSpecName "kube-api-access-wqq9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:14.768761 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:14.768725    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqq9m\" (UniqueName: \"kubernetes.io/projected/b991fb45-8e0f-4fba-93e1-cf0086a825fe-kube-api-access-wqq9m\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:14.768761 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:14.768756    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b991fb45-8e0f-4fba-93e1-cf0086a825fe-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:15.407166 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:15.407139    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"
Oct 30 19:53:15.407166 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:15.407162    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf" event={"ID":"b991fb45-8e0f-4fba-93e1-cf0086a825fe","Type":"ContainerDied","Data":"4bbb01b7c5318e500536765773461d1e65a431c2b1f2878ed89cc0ee9b906786"}
Oct 30 19:53:15.407375 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:15.407204    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bbb01b7c5318e500536765773461d1e65a431c2b1f2878ed89cc0ee9b906786"
Oct 30 19:53:15.546982 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:15.546955    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:15.560528 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:15.560499    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:16.426095 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:16.426066    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:53:19.438683 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.438646    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"]
Oct 30 19:53:19.439314 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.439298    4823 cpu_manager.go:414] "containerMap: removing container" podUID="b991fb45-8e0f-4fba-93e1-cf0086a825fe" containerName="revoke-keys"
Oct 30 19:53:19.439314 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.439315    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b991fb45-8e0f-4fba-93e1-cf0086a825fe" containerName="revoke-keys"
Oct 30 19:53:19.439440 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.439430    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="b991fb45-8e0f-4fba-93e1-cf0086a825fe" containerName="revoke-keys"
Oct 30 19:53:19.443088 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.443067    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"
Oct 30 19:53:19.455152 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.455124    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:53:19.455254 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.455175    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:53:19.455573 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.455554    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:53:19.457416 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.457382    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:53:19.462951 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.462909    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"]
Oct 30 19:53:19.544150 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.544111    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f21689d-3d9f-43f7-8927-bae70428d5e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a68ad3c-jkpch\" (UID: \"4f21689d-3d9f-43f7-8927-bae70428d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"
Oct 30 19:53:19.544150 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.544156    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f21689d-3d9f-43f7-8927-bae70428d5e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a68ad3c-jkpch\" (UID: \"4f21689d-3d9f-43f7-8927-bae70428d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"
Oct 30 19:53:19.544363 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.544295    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlcxv\" (UniqueName: \"kubernetes.io/projected/4f21689d-3d9f-43f7-8927-bae70428d5e9-kube-api-access-dlcxv\") pod \"maas-api-revoke-sub-5a68ad3c-jkpch\" (UID: \"4f21689d-3d9f-43f7-8927-bae70428d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"
Oct 30 19:53:19.645432 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.645395    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f21689d-3d9f-43f7-8927-bae70428d5e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a68ad3c-jkpch\" (UID: \"4f21689d-3d9f-43f7-8927-bae70428d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"
Oct 30 19:53:19.645634 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.645513    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f21689d-3d9f-43f7-8927-bae70428d5e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a68ad3c-jkpch\" (UID: \"4f21689d-3d9f-43f7-8927-bae70428d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"
Oct 30 19:53:19.645698 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.645660    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlcxv\" (UniqueName: \"kubernetes.io/projected/4f21689d-3d9f-43f7-8927-bae70428d5e9-kube-api-access-dlcxv\") pod \"maas-api-revoke-sub-5a68ad3c-jkpch\" (UID: \"4f21689d-3d9f-43f7-8927-bae70428d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"
Oct 30 19:53:19.646293 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.646270    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f21689d-3d9f-43f7-8927-bae70428d5e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a68ad3c-jkpch\" (UID: \"4f21689d-3d9f-43f7-8927-bae70428d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"
Oct 30 19:53:19.647906 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.647884    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f21689d-3d9f-43f7-8927-bae70428d5e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a68ad3c-jkpch\" (UID: \"4f21689d-3d9f-43f7-8927-bae70428d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"
Oct 30 19:53:19.657489 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.657466    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlcxv\" (UniqueName: \"kubernetes.io/projected/4f21689d-3d9f-43f7-8927-bae70428d5e9-kube-api-access-dlcxv\") pod \"maas-api-revoke-sub-5a68ad3c-jkpch\" (UID: \"4f21689d-3d9f-43f7-8927-bae70428d5e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"
Oct 30 19:53:19.757506 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:19.757430    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"
Oct 30 19:53:20.049412 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:20.049377    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"]
Oct 30 19:53:20.050856 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:53:20.050827    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f21689d_3d9f_43f7_8927_bae70428d5e9.slice/crio-5b837b7605404336a9911e0d30ba168307a7e0860a1403322a218822510b7e44 WatchSource:0}: Error finding container 5b837b7605404336a9911e0d30ba168307a7e0860a1403322a218822510b7e44: Status 404 returned error can't find the container with id 5b837b7605404336a9911e0d30ba168307a7e0860a1403322a218822510b7e44
Oct 30 19:53:20.447705 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:20.447667    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch" event={"ID":"4f21689d-3d9f-43f7-8927-bae70428d5e9","Type":"ContainerStarted","Data":"12da6bfeb5e2bfa9145527e1abd38c268ea9f9e99edc41135057ee62a06b3162"}
Oct 30 19:53:20.448123 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:20.447766    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch" event={"ID":"4f21689d-3d9f-43f7-8927-bae70428d5e9","Type":"ContainerStarted","Data":"5b837b7605404336a9911e0d30ba168307a7e0860a1403322a218822510b7e44"}
Oct 30 19:53:20.470779 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:20.470725    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch" podStartSLOduration=1.470710796 podStartE2EDuration="1.470710796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:20.466365178 +0000 UTC m=+1400.358979526" watchObservedRunningTime="2026-10-30 19:53:20.470710796 +0000 UTC m=+1400.363325145"
Oct 30 19:53:21.456177 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:21.456140    4823 generic.go:356] "Generic (PLEG): container finished" podID="4f21689d-3d9f-43f7-8927-bae70428d5e9" containerID="12da6bfeb5e2bfa9145527e1abd38c268ea9f9e99edc41135057ee62a06b3162" exitCode=0
Oct 30 19:53:21.456636 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:21.456221    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch" event={"ID":"4f21689d-3d9f-43f7-8927-bae70428d5e9","Type":"ContainerDied","Data":"12da6bfeb5e2bfa9145527e1abd38c268ea9f9e99edc41135057ee62a06b3162"}
Oct 30 19:53:22.644959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:22.644935    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"
Oct 30 19:53:22.682330 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:22.682297    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f21689d-3d9f-43f7-8927-bae70428d5e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f21689d-3d9f-43f7-8927-bae70428d5e9-maas-api-openshift-ca\") pod \"4f21689d-3d9f-43f7-8927-bae70428d5e9\" (UID: \"4f21689d-3d9f-43f7-8927-bae70428d5e9\") "
Oct 30 19:53:22.682509 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:22.682442    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f21689d-3d9f-43f7-8927-bae70428d5e9-kube-api-access-dlcxv\" (UniqueName: \"kubernetes.io/projected/4f21689d-3d9f-43f7-8927-bae70428d5e9-kube-api-access-dlcxv\") pod \"4f21689d-3d9f-43f7-8927-bae70428d5e9\" (UID: \"4f21689d-3d9f-43f7-8927-bae70428d5e9\") "
Oct 30 19:53:22.682590 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:22.682529    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f21689d-3d9f-43f7-8927-bae70428d5e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f21689d-3d9f-43f7-8927-bae70428d5e9-maas-api-xks-ca\") pod \"4f21689d-3d9f-43f7-8927-bae70428d5e9\" (UID: \"4f21689d-3d9f-43f7-8927-bae70428d5e9\") "
Oct 30 19:53:22.682783 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:22.682751    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f21689d-3d9f-43f7-8927-bae70428d5e9-maas-api-openshift-ca" pod "4f21689d-3d9f-43f7-8927-bae70428d5e9" (UID: "4f21689d-3d9f-43f7-8927-bae70428d5e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:22.683249 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:22.683210    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f21689d-3d9f-43f7-8927-bae70428d5e9-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:22.685308 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:22.685286    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f21689d-3d9f-43f7-8927-bae70428d5e9-kube-api-access-dlcxv" pod "4f21689d-3d9f-43f7-8927-bae70428d5e9" (UID: "4f21689d-3d9f-43f7-8927-bae70428d5e9"). InnerVolumeSpecName "kube-api-access-dlcxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:22.689977 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:22.689951    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f21689d-3d9f-43f7-8927-bae70428d5e9-maas-api-xks-ca" pod "4f21689d-3d9f-43f7-8927-bae70428d5e9" (UID: "4f21689d-3d9f-43f7-8927-bae70428d5e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:22.784104 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:22.784022    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlcxv\" (UniqueName: \"kubernetes.io/projected/4f21689d-3d9f-43f7-8927-bae70428d5e9-kube-api-access-dlcxv\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:22.784104 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:22.784049    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f21689d-3d9f-43f7-8927-bae70428d5e9-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:23.493918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:23.490704    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"
Oct 30 19:53:23.493918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:23.490722    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch" event={"ID":"4f21689d-3d9f-43f7-8927-bae70428d5e9","Type":"ContainerDied","Data":"5b837b7605404336a9911e0d30ba168307a7e0860a1403322a218822510b7e44"}
Oct 30 19:53:23.493918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:23.490783    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b837b7605404336a9911e0d30ba168307a7e0860a1403322a218822510b7e44"
Oct 30 19:53:25.808526 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:25.808492    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"]
Oct 30 19:53:25.809210 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:25.809191    4823 cpu_manager.go:414] "containerMap: removing container" podUID="4f21689d-3d9f-43f7-8927-bae70428d5e9" containerName="revoke-keys"
Oct 30 19:53:25.809210 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:25.809209    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f21689d-3d9f-43f7-8927-bae70428d5e9" containerName="revoke-keys"
Oct 30 19:53:25.809350 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:25.809332    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f21689d-3d9f-43f7-8927-bae70428d5e9" containerName="revoke-keys"
Oct 30 19:53:25.813239 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:25.813217    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:25.834227 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:25.834200    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"]
Oct 30 19:53:25.934916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:25.934851    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:25.935062 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:25.934975    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:25.935062 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:25.935029    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8828115-a9ef-41dc-bf28-f5dfb3131377-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:25.935163 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:25.935063    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:25.935213 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:25.935199    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:25.935249 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:25.935220    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:25.935282 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:25.935257    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbbc8\" (UniqueName: \"kubernetes.io/projected/e8828115-a9ef-41dc-bf28-f5dfb3131377-kube-api-access-gbbc8\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:25.935381 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:25.935360    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:25.935419 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:25.935393    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.037037 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.037003    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbbc8\" (UniqueName: \"kubernetes.io/projected/e8828115-a9ef-41dc-bf28-f5dfb3131377-kube-api-access-gbbc8\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.037183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.037053    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.037183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.037083    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.037183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.037165    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.037302 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.037211    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.037302 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.037271    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8828115-a9ef-41dc-bf28-f5dfb3131377-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.037391 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.037314    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.037443 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.037417    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.037492 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.037445    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.037569 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.037543    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.037749 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.037725    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.037856 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.037750    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.037856 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.037778    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.038226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.038195    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8828115-a9ef-41dc-bf28-f5dfb3131377-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.039612 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.039588    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.039990 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.039972    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.062502 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.062449    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.066116 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.066090    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbbc8\" (UniqueName: \"kubernetes.io/projected/e8828115-a9ef-41dc-bf28-f5dfb3131377-kube-api-access-gbbc8\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.132427 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.132385    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:26.241440 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.241409    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"]
Oct 30 19:53:26.248434 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.248411    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"
Oct 30 19:53:26.256777 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.255416    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:53:26.256777 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.256541    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:53:26.256777 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.256638    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:53:26.256777 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.256738    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:53:26.266816 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.263234    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"]
Oct 30 19:53:26.340941 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.340834    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01638ce8-8f1a-41ae-972c-503676f04eac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cd1db51-mvltf\" (UID: \"01638ce8-8f1a-41ae-972c-503676f04eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"
Oct 30 19:53:26.340941 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.340904    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjk4d\" (UniqueName: \"kubernetes.io/projected/01638ce8-8f1a-41ae-972c-503676f04eac-kube-api-access-bjk4d\") pod \"maas-api-revoke-sub-0cd1db51-mvltf\" (UID: \"01638ce8-8f1a-41ae-972c-503676f04eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"
Oct 30 19:53:26.341132 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.341114    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01638ce8-8f1a-41ae-972c-503676f04eac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cd1db51-mvltf\" (UID: \"01638ce8-8f1a-41ae-972c-503676f04eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"
Oct 30 19:53:26.443165 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.443130    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01638ce8-8f1a-41ae-972c-503676f04eac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cd1db51-mvltf\" (UID: \"01638ce8-8f1a-41ae-972c-503676f04eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"
Oct 30 19:53:26.443165 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.443167    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjk4d\" (UniqueName: \"kubernetes.io/projected/01638ce8-8f1a-41ae-972c-503676f04eac-kube-api-access-bjk4d\") pod \"maas-api-revoke-sub-0cd1db51-mvltf\" (UID: \"01638ce8-8f1a-41ae-972c-503676f04eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"
Oct 30 19:53:26.443430 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.443239    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01638ce8-8f1a-41ae-972c-503676f04eac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cd1db51-mvltf\" (UID: \"01638ce8-8f1a-41ae-972c-503676f04eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"
Oct 30 19:53:26.444055 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.444023    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01638ce8-8f1a-41ae-972c-503676f04eac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cd1db51-mvltf\" (UID: \"01638ce8-8f1a-41ae-972c-503676f04eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"
Oct 30 19:53:26.446499 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.446319    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01638ce8-8f1a-41ae-972c-503676f04eac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cd1db51-mvltf\" (UID: \"01638ce8-8f1a-41ae-972c-503676f04eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"
Oct 30 19:53:26.449872 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:53:26.449826    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8828115_a9ef_41dc_bf28_f5dfb3131377.slice/crio-161f8146c8407bbef8511f0fc8b429582baac55bdac0137096c39ba88033fe31 WatchSource:0}: Error finding container 161f8146c8407bbef8511f0fc8b429582baac55bdac0137096c39ba88033fe31: Status 404 returned error can't find the container with id 161f8146c8407bbef8511f0fc8b429582baac55bdac0137096c39ba88033fe31
Oct 30 19:53:26.450065 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.450047    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"]
Oct 30 19:53:26.452658 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.452620    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:26.452766 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.452697    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:26.452766 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.452738    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:26.466697 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.466671    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjk4d\" (UniqueName: \"kubernetes.io/projected/01638ce8-8f1a-41ae-972c-503676f04eac-kube-api-access-bjk4d\") pod \"maas-api-revoke-sub-0cd1db51-mvltf\" (UID: \"01638ce8-8f1a-41ae-972c-503676f04eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"
Oct 30 19:53:26.532401 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.532364    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f" event={"ID":"e8828115-a9ef-41dc-bf28-f5dfb3131377","Type":"ContainerStarted","Data":"2929f0c963c05b52d87d5fd721ee1982e17c0d6757933f1130cb571f6fb4c445"}
Oct 30 19:53:26.532565 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.532434    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f" event={"ID":"e8828115-a9ef-41dc-bf28-f5dfb3131377","Type":"ContainerStarted","Data":"161f8146c8407bbef8511f0fc8b429582baac55bdac0137096c39ba88033fe31"}
Oct 30 19:53:26.569475 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.569408    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f" podStartSLOduration=1.5693879160000002 podStartE2EDuration="1.569387916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:26.564263995 +0000 UTC m=+1406.456878321" watchObservedRunningTime="2026-10-30 19:53:26.569387916 +0000 UTC m=+1406.462002243"
Oct 30 19:53:26.608489 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.608458    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"
Oct 30 19:53:26.919561 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:26.919533    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"]
Oct 30 19:53:26.922003 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:53:26.920781    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01638ce8_8f1a_41ae_972c_503676f04eac.slice/crio-f7eca9f40cb54b78d155ca87eff3e3f53ba3f72563458ce15bbddd98006048bb WatchSource:0}: Error finding container f7eca9f40cb54b78d155ca87eff3e3f53ba3f72563458ce15bbddd98006048bb: Status 404 returned error can't find the container with id f7eca9f40cb54b78d155ca87eff3e3f53ba3f72563458ce15bbddd98006048bb
Oct 30 19:53:27.132829 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:27.132795    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:27.134822 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:27.134793    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:53:27.134952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:27.134850    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f" podUID="e8828115-a9ef-41dc-bf28-f5dfb3131377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:53:27.541502 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:27.541413    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf" event={"ID":"01638ce8-8f1a-41ae-972c-503676f04eac","Type":"ContainerStarted","Data":"564e8a686e8f68ccb1773e823c54b3acd2d58b9959e342fa56eedab3ccf4d1bd"}
Oct 30 19:53:27.541502 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:27.541469    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf" event={"ID":"01638ce8-8f1a-41ae-972c-503676f04eac","Type":"ContainerStarted","Data":"f7eca9f40cb54b78d155ca87eff3e3f53ba3f72563458ce15bbddd98006048bb"}
Oct 30 19:53:27.573887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:27.563514    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf" podStartSLOduration=1.563492179 podStartE2EDuration="1.563492179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:27.560259317 +0000 UTC m=+1407.452873644" watchObservedRunningTime="2026-10-30 19:53:27.563492179 +0000 UTC m=+1407.456106504"
Oct 30 19:53:28.133276 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:28.133244    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:53:28.133646 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:28.133298    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f" podUID="e8828115-a9ef-41dc-bf28-f5dfb3131377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:53:28.550471 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:28.550388    4823 generic.go:356] "Generic (PLEG): container finished" podID="01638ce8-8f1a-41ae-972c-503676f04eac" containerID="564e8a686e8f68ccb1773e823c54b3acd2d58b9959e342fa56eedab3ccf4d1bd" exitCode=0
Oct 30 19:53:28.550618 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:28.550474    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf" event={"ID":"01638ce8-8f1a-41ae-972c-503676f04eac","Type":"ContainerDied","Data":"564e8a686e8f68ccb1773e823c54b3acd2d58b9959e342fa56eedab3ccf4d1bd"}
Oct 30 19:53:29.133088 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:29.133055    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:53:29.133264 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:29.133114    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f" podUID="e8828115-a9ef-41dc-bf28-f5dfb3131377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:53:29.793619 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:29.793590    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"
Oct 30 19:53:29.916432 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:29.916404    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01638ce8-8f1a-41ae-972c-503676f04eac-kube-api-access-bjk4d\" (UniqueName: \"kubernetes.io/projected/01638ce8-8f1a-41ae-972c-503676f04eac-kube-api-access-bjk4d\") pod \"01638ce8-8f1a-41ae-972c-503676f04eac\" (UID: \"01638ce8-8f1a-41ae-972c-503676f04eac\") "
Oct 30 19:53:29.916643 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:29.916487    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01638ce8-8f1a-41ae-972c-503676f04eac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01638ce8-8f1a-41ae-972c-503676f04eac-maas-api-openshift-ca\") pod \"01638ce8-8f1a-41ae-972c-503676f04eac\" (UID: \"01638ce8-8f1a-41ae-972c-503676f04eac\") "
Oct 30 19:53:29.916643 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:29.916565    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01638ce8-8f1a-41ae-972c-503676f04eac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01638ce8-8f1a-41ae-972c-503676f04eac-maas-api-xks-ca\") pod \"01638ce8-8f1a-41ae-972c-503676f04eac\" (UID: \"01638ce8-8f1a-41ae-972c-503676f04eac\") "
Oct 30 19:53:29.916850 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:29.916825    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01638ce8-8f1a-41ae-972c-503676f04eac-maas-api-openshift-ca" pod "01638ce8-8f1a-41ae-972c-503676f04eac" (UID: "01638ce8-8f1a-41ae-972c-503676f04eac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:29.917094 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:29.917077    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01638ce8-8f1a-41ae-972c-503676f04eac-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:29.918738 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:29.918701    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01638ce8-8f1a-41ae-972c-503676f04eac-kube-api-access-bjk4d" pod "01638ce8-8f1a-41ae-972c-503676f04eac" (UID: "01638ce8-8f1a-41ae-972c-503676f04eac"). InnerVolumeSpecName "kube-api-access-bjk4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:29.918851 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:29.918756    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01638ce8-8f1a-41ae-972c-503676f04eac-maas-api-xks-ca" pod "01638ce8-8f1a-41ae-972c-503676f04eac" (UID: "01638ce8-8f1a-41ae-972c-503676f04eac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:30.018592 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:30.018544    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjk4d\" (UniqueName: \"kubernetes.io/projected/01638ce8-8f1a-41ae-972c-503676f04eac-kube-api-access-bjk4d\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:30.018592 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:30.018576    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01638ce8-8f1a-41ae-972c-503676f04eac-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:30.133120 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:30.133084    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:53:30.133294 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:30.133148    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f" podUID="e8828115-a9ef-41dc-bf28-f5dfb3131377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:53:30.587126 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:30.587046    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"
Oct 30 19:53:30.587126 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:30.587078    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf" event={"ID":"01638ce8-8f1a-41ae-972c-503676f04eac","Type":"ContainerDied","Data":"f7eca9f40cb54b78d155ca87eff3e3f53ba3f72563458ce15bbddd98006048bb"}
Oct 30 19:53:30.587313 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:30.587131    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7eca9f40cb54b78d155ca87eff3e3f53ba3f72563458ce15bbddd98006048bb"
Oct 30 19:53:31.134018 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:31.133976    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:53:31.134467 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:31.134052    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f" podUID="e8828115-a9ef-41dc-bf28-f5dfb3131377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:53:32.015329 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:53:32.015276    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:53:32.133292 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:32.133251    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:53:32.133496 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:32.133323    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f" podUID="e8828115-a9ef-41dc-bf28-f5dfb3131377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:53:33.129428 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:33.129393    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:53:33.133101 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:33.133074    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:53:33.133213 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:33.133136    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f" podUID="e8828115-a9ef-41dc-bf28-f5dfb3131377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:53:34.138294 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:34.138260    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:34.138839 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:34.138815    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:34.139492 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:34.139474    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 19:53:36.050651 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:36.050605    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:36.203491 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:36.203445    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:36.238251 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:36.227385    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:53:36.429885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:36.421955    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:36.434251 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:36.434206    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:36.557469 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:36.555947    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": context deadline exceeded"
Oct 30 19:53:36.695938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:36.695002    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:40.597734 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.597694    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"]
Oct 30 19:53:40.598800 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.598776    4823 cpu_manager.go:414] "containerMap: removing container" podUID="01638ce8-8f1a-41ae-972c-503676f04eac" containerName="revoke-keys"
Oct 30 19:53:40.598852 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.598803    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01638ce8-8f1a-41ae-972c-503676f04eac" containerName="revoke-keys"
Oct 30 19:53:40.599061 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.599044    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="01638ce8-8f1a-41ae-972c-503676f04eac" containerName="revoke-keys"
Oct 30 19:53:40.604896 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.604872    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.617478 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.617448    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"]
Oct 30 19:53:40.769832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.769739    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.770085 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.769894    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd5z4\" (UniqueName: \"kubernetes.io/projected/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-kube-api-access-rd5z4\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.770085 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.769955    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.770190 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.770119    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.770190 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.770180    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.770297 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.770279    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.770342 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.770314    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.770501 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.770486    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.770554 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.770541    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.871684 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.871638    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.871914 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.871689    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.871914 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.871714    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd5z4\" (UniqueName: \"kubernetes.io/projected/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-kube-api-access-rd5z4\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.871914 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.871776    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.872095 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.871915    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.872095 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.871953    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.872095 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.872024    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.872095 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.872054    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.872310 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.872224    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.872310 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.872303    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.872470 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.872437    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.872541 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.872478    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.872601 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.872584    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.872637 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.872614    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.874573 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.874546    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.874779 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.874761    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.881030 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.881007    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"]
Oct 30 19:53:40.887100 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.887082    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"
Oct 30 19:53:40.892003 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.891891    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.892003 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.891960    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd5z4\" (UniqueName: \"kubernetes.io/projected/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-kube-api-access-rd5z4\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.894404 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.894381    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:53:40.894520 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.894455    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:53:40.894520 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.894478    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:53:40.894622 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.894533    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:53:40.900694 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.900665    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"]
Oct 30 19:53:40.931276 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.931248    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:40.973637 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.973606    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-263ad6b7-2t7bh\" (UID: \"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"
Oct 30 19:53:40.973790 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.973692    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-263ad6b7-2t7bh\" (UID: \"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"
Oct 30 19:53:40.973790 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:40.973764    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gbrg\" (UniqueName: \"kubernetes.io/projected/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-kube-api-access-8gbrg\") pod \"maas-api-revoke-sub-263ad6b7-2t7bh\" (UID: \"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"
Oct 30 19:53:41.074909 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.074845    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gbrg\" (UniqueName: \"kubernetes.io/projected/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-kube-api-access-8gbrg\") pod \"maas-api-revoke-sub-263ad6b7-2t7bh\" (UID: \"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"
Oct 30 19:53:41.075100 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.075032    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-263ad6b7-2t7bh\" (UID: \"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"
Oct 30 19:53:41.075169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.075112    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-263ad6b7-2t7bh\" (UID: \"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"
Oct 30 19:53:41.075772 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.075745    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-263ad6b7-2t7bh\" (UID: \"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"
Oct 30 19:53:41.078152 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.078123    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-263ad6b7-2t7bh\" (UID: \"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"
Oct 30 19:53:41.095189 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.095166    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gbrg\" (UniqueName: \"kubernetes.io/projected/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-kube-api-access-8gbrg\") pod \"maas-api-revoke-sub-263ad6b7-2t7bh\" (UID: \"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"
Oct 30 19:53:41.225089 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:53:41.225058    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc11d6a_c12d_4dab_8356_a8a05ee82f37.slice/crio-d21a7525f2346526b3a189a9079cabc06a5de91b0f8637aec0cf4ac0677f79da WatchSource:0}: Error finding container d21a7525f2346526b3a189a9079cabc06a5de91b0f8637aec0cf4ac0677f79da: Status 404 returned error can't find the container with id d21a7525f2346526b3a189a9079cabc06a5de91b0f8637aec0cf4ac0677f79da
Oct 30 19:53:41.225225 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.225103    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"
Oct 30 19:53:41.225784 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.225760    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"]
Oct 30 19:53:41.227982 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.227956    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:41.228060 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.228013    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:41.228060 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.228041    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:41.336635 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.336601    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"]
Oct 30 19:53:41.343474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.342552    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.361534 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.361499    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"]
Oct 30 19:53:41.480614 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.480527    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sdm5\" (UniqueName: \"kubernetes.io/projected/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-kube-api-access-9sdm5\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.480614 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.480597    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.480880 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.480779    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.480880 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.480807    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.481004 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.480997    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.481062 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.481017    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.481062 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.481053    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.481175 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.481140    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.481235 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.481180    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.518900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.518870    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"]
Oct 30 19:53:41.519341 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:53:41.519315    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9c3d3f_fe44_454a_a2d3_11c3d8618d8f.slice/crio-bc61b54a2d34c511c45bca7b9e7cd16826f50cade4fc49c312eccabbd84199b4 WatchSource:0}: Error finding container bc61b54a2d34c511c45bca7b9e7cd16826f50cade4fc49c312eccabbd84199b4: Status 404 returned error can't find the container with id bc61b54a2d34c511c45bca7b9e7cd16826f50cade4fc49c312eccabbd84199b4
Oct 30 19:53:41.583057 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.583015    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.583178 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.583062    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.583178 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.583111    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.583178 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.583169    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.583354 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.583205    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.583354 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.583246    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdm5\" (UniqueName: \"kubernetes.io/projected/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-kube-api-access-9sdm5\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.583354 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.583339    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.583509 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.583455    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.583509 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.583490    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.583620 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.583558    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.584109 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.583693    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.584109 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.583947    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.584109 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.584005    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.584510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.584490    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.585749 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.585726    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.586609 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.586587    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.595824 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.595798    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.604088 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.604068    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sdm5\" (UniqueName: \"kubernetes.io/projected/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-kube-api-access-9sdm5\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.666220 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.666185    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:41.898725 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.898382    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv" event={"ID":"3bc11d6a-c12d-4dab-8356-a8a05ee82f37","Type":"ContainerStarted","Data":"2e37ea1ce0eeeb34712b575c3728d81eb5656a9a36f6a3e8b9c5d387e45779cc"}
Oct 30 19:53:41.898725 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.898448    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv" event={"ID":"3bc11d6a-c12d-4dab-8356-a8a05ee82f37","Type":"ContainerStarted","Data":"d21a7525f2346526b3a189a9079cabc06a5de91b0f8637aec0cf4ac0677f79da"}
Oct 30 19:53:41.916850 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.916152    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh" event={"ID":"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f","Type":"ContainerStarted","Data":"bc61b54a2d34c511c45bca7b9e7cd16826f50cade4fc49c312eccabbd84199b4"}
Oct 30 19:53:41.934050 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.933973    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:41.937395 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.934919    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv" podStartSLOduration=1.934906018 podStartE2EDuration="1.934906018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:41.934597568 +0000 UTC m=+1421.827211893" watchObservedRunningTime="2026-10-30 19:53:41.934906018 +0000 UTC m=+1421.827520341"
Oct 30 19:53:41.981695 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.981624    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh" podStartSLOduration=1.9816018610000001 podStartE2EDuration="1.981601861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:41.976297816 +0000 UTC m=+1421.868912141" watchObservedRunningTime="2026-10-30 19:53:41.981601861 +0000 UTC m=+1421.874216186"
Oct 30 19:53:41.989014 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:53:41.988941    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode473bfd6_107c_4cc1_a9f8_8f8e546b743b.slice/crio-96701ee7a298a03f516a8d273fd7ee2869e8e977a913c45d1c37771fec37a9f1 WatchSource:0}: Error finding container 96701ee7a298a03f516a8d273fd7ee2869e8e977a913c45d1c37771fec37a9f1: Status 404 returned error can't find the container with id 96701ee7a298a03f516a8d273fd7ee2869e8e977a913c45d1c37771fec37a9f1
Oct 30 19:53:41.989567 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.989536    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"]
Oct 30 19:53:41.992004 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.991973    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:41.992084 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.992033    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:41.992084 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:41.992062    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:42.719086 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:42.719048    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"]
Oct 30 19:53:42.725341 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:42.725322    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"
Oct 30 19:53:42.734824 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:42.734798    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"]
Oct 30 19:53:42.903610 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:42.903573    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400327da-db74-4cce-be08-c75a73263374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c330f80-j6f85\" (UID: \"400327da-db74-4cce-be08-c75a73263374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"
Oct 30 19:53:42.903799 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:42.903706    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400327da-db74-4cce-be08-c75a73263374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c330f80-j6f85\" (UID: \"400327da-db74-4cce-be08-c75a73263374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"
Oct 30 19:53:42.903849 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:42.903799    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz5bn\" (UniqueName: \"kubernetes.io/projected/400327da-db74-4cce-be08-c75a73263374-kube-api-access-jz5bn\") pod \"maas-api-revoke-sub-6c330f80-j6f85\" (UID: \"400327da-db74-4cce-be08-c75a73263374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"
Oct 30 19:53:42.934882 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:42.932654    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb" event={"ID":"e473bfd6-107c-4cc1-a9f8-8f8e546b743b","Type":"ContainerStarted","Data":"27676435341d0f90cb433f0fa7d9007d60ceba3f562c8544bd556ab4c19c3d65"}
Oct 30 19:53:42.934882 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:42.932731    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb" event={"ID":"e473bfd6-107c-4cc1-a9f8-8f8e546b743b","Type":"ContainerStarted","Data":"96701ee7a298a03f516a8d273fd7ee2869e8e977a913c45d1c37771fec37a9f1"}
Oct 30 19:53:42.934882 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:42.934747    4823 generic.go:356] "Generic (PLEG): container finished" podID="2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f" containerID="75f5cfb3ed02294fa68421889733393c38cbc7259e2e7499ec83b5757611add1" exitCode=0
Oct 30 19:53:42.935113 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:42.934888    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh" event={"ID":"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f","Type":"ContainerDied","Data":"75f5cfb3ed02294fa68421889733393c38cbc7259e2e7499ec83b5757611add1"}
Oct 30 19:53:42.935385 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:42.935364    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:53:42.935434 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:42.935415    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv" podUID="3bc11d6a-c12d-4dab-8356-a8a05ee82f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:53:43.007278 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:43.007109    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb" podStartSLOduration=2.007082985 podStartE2EDuration="2.007082985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:42.972219034 +0000 UTC m=+1422.864833351" watchObservedRunningTime="2026-10-30 19:53:43.007082985 +0000 UTC m=+1422.899697310"
Oct 30 19:53:43.008466 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:43.008436    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400327da-db74-4cce-be08-c75a73263374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c330f80-j6f85\" (UID: \"400327da-db74-4cce-be08-c75a73263374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"
Oct 30 19:53:43.008616 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:43.008564    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400327da-db74-4cce-be08-c75a73263374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c330f80-j6f85\" (UID: \"400327da-db74-4cce-be08-c75a73263374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"
Oct 30 19:53:43.008779 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:43.008762    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz5bn\" (UniqueName: \"kubernetes.io/projected/400327da-db74-4cce-be08-c75a73263374-kube-api-access-jz5bn\") pod \"maas-api-revoke-sub-6c330f80-j6f85\" (UID: \"400327da-db74-4cce-be08-c75a73263374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"
Oct 30 19:53:43.009402 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:43.009374    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400327da-db74-4cce-be08-c75a73263374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c330f80-j6f85\" (UID: \"400327da-db74-4cce-be08-c75a73263374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"
Oct 30 19:53:43.011636 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:43.011616    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400327da-db74-4cce-be08-c75a73263374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c330f80-j6f85\" (UID: \"400327da-db74-4cce-be08-c75a73263374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"
Oct 30 19:53:43.023296 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:43.022822    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz5bn\" (UniqueName: \"kubernetes.io/projected/400327da-db74-4cce-be08-c75a73263374-kube-api-access-jz5bn\") pod \"maas-api-revoke-sub-6c330f80-j6f85\" (UID: \"400327da-db74-4cce-be08-c75a73263374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"
Oct 30 19:53:43.043547 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:43.043522    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"
Oct 30 19:53:43.533471 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:43.533437    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"]
Oct 30 19:53:43.534046 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:53:43.534020    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400327da_db74_4cce_be08_c75a73263374.slice/crio-eed6093d22bd6a6c71c2b466ce455635a8b183b093b0f3ed9e42a9eab5932848 WatchSource:0}: Error finding container eed6093d22bd6a6c71c2b466ce455635a8b183b093b0f3ed9e42a9eab5932848: Status 404 returned error can't find the container with id eed6093d22bd6a6c71c2b466ce455635a8b183b093b0f3ed9e42a9eab5932848
Oct 30 19:53:43.686924 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:43.678924    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:43.686924 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:43.683983    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:53:43.686924 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:43.684043    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb" podUID="e473bfd6-107c-4cc1-a9f8-8f8e546b743b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 19:53:43.932264 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:43.932230    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:53:43.932624 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:43.932285    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv" podUID="3bc11d6a-c12d-4dab-8356-a8a05ee82f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:53:43.944551 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:43.944520    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85" event={"ID":"400327da-db74-4cce-be08-c75a73263374","Type":"ContainerStarted","Data":"eed6093d22bd6a6c71c2b466ce455635a8b183b093b0f3ed9e42a9eab5932848"}
Oct 30 19:53:44.351435 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.351411    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"
Oct 30 19:53:44.543146 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.543110    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-kube-api-access-8gbrg\" (UniqueName: \"kubernetes.io/projected/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-kube-api-access-8gbrg\") pod \"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f\" (UID: \"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f\") "
Oct 30 19:53:44.543345 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.543158    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-maas-api-openshift-ca\") pod \"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f\" (UID: \"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f\") "
Oct 30 19:53:44.543345 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.543205    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-maas-api-xks-ca\") pod \"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f\" (UID: \"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f\") "
Oct 30 19:53:44.543538 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.543511    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-maas-api-openshift-ca" pod "2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f" (UID: "2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:44.543732 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.543716    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:44.545403 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.545379    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-maas-api-xks-ca" pod "2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f" (UID: "2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:44.545403 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.545395    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-kube-api-access-8gbrg" pod "2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f" (UID: "2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f"). InnerVolumeSpecName "kube-api-access-8gbrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:44.645515 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.645484    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:44.645690 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.645679    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gbrg\" (UniqueName: \"kubernetes.io/projected/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f-kube-api-access-8gbrg\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:44.687549 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.687514    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:53:44.688286 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.688250    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb" podUID="e473bfd6-107c-4cc1-a9f8-8f8e546b743b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 19:53:44.931991 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.931906    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:53:44.931991 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.931977    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv" podUID="3bc11d6a-c12d-4dab-8356-a8a05ee82f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:53:44.975885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.974604    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh" event={"ID":"2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f","Type":"ContainerDied","Data":"bc61b54a2d34c511c45bca7b9e7cd16826f50cade4fc49c312eccabbd84199b4"}
Oct 30 19:53:44.975885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.974673    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc61b54a2d34c511c45bca7b9e7cd16826f50cade4fc49c312eccabbd84199b4"
Oct 30 19:53:44.975885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:44.974755    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"
Oct 30 19:53:45.005889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:45.004442    4823 generic.go:356] "Generic (PLEG): container finished" podID="400327da-db74-4cce-be08-c75a73263374" containerID="87fcb51cc178d26a0ad86bbb57ce63d2955b652a73fafc1a196cd963b7bc2dab" exitCode=0
Oct 30 19:53:45.007085 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:45.006962    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85" event={"ID":"400327da-db74-4cce-be08-c75a73263374","Type":"ContainerDied","Data":"87fcb51cc178d26a0ad86bbb57ce63d2955b652a73fafc1a196cd963b7bc2dab"}
Oct 30 19:53:45.698935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:45.685317    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:53:45.698935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:45.685436    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb" podUID="e473bfd6-107c-4cc1-a9f8-8f8e546b743b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 19:53:45.932818 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:45.932217    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:53:45.932818 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:45.932280    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv" podUID="3bc11d6a-c12d-4dab-8356-a8a05ee82f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:53:46.485102 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:46.485069    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"
Oct 30 19:53:46.628481 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:46.628437    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400327da-db74-4cce-be08-c75a73263374-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400327da-db74-4cce-be08-c75a73263374-maas-api-xks-ca\") pod \"400327da-db74-4cce-be08-c75a73263374\" (UID: \"400327da-db74-4cce-be08-c75a73263374\") "
Oct 30 19:53:46.628640 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:46.628507    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/400327da-db74-4cce-be08-c75a73263374-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400327da-db74-4cce-be08-c75a73263374-maas-api-openshift-ca\") pod \"400327da-db74-4cce-be08-c75a73263374\" (UID: \"400327da-db74-4cce-be08-c75a73263374\") "
Oct 30 19:53:46.628640 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:46.628592    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400327da-db74-4cce-be08-c75a73263374-kube-api-access-jz5bn\" (UniqueName: \"kubernetes.io/projected/400327da-db74-4cce-be08-c75a73263374-kube-api-access-jz5bn\") pod \"400327da-db74-4cce-be08-c75a73263374\" (UID: \"400327da-db74-4cce-be08-c75a73263374\") "
Oct 30 19:53:46.629123 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:46.629070    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400327da-db74-4cce-be08-c75a73263374-maas-api-openshift-ca" pod "400327da-db74-4cce-be08-c75a73263374" (UID: "400327da-db74-4cce-be08-c75a73263374"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:46.629466 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:46.629434    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400327da-db74-4cce-be08-c75a73263374-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:46.632894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:46.631087    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400327da-db74-4cce-be08-c75a73263374-maas-api-xks-ca" pod "400327da-db74-4cce-be08-c75a73263374" (UID: "400327da-db74-4cce-be08-c75a73263374"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:46.632894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:46.631387    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400327da-db74-4cce-be08-c75a73263374-kube-api-access-jz5bn" pod "400327da-db74-4cce-be08-c75a73263374" (UID: "400327da-db74-4cce-be08-c75a73263374"). InnerVolumeSpecName "kube-api-access-jz5bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:46.677058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:46.676965    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:53:46.677058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:46.677032    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb" podUID="e473bfd6-107c-4cc1-a9f8-8f8e546b743b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 19:53:46.730972 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:46.730940    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400327da-db74-4cce-be08-c75a73263374-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:46.730972 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:46.730971    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz5bn\" (UniqueName: \"kubernetes.io/projected/400327da-db74-4cce-be08-c75a73263374-kube-api-access-jz5bn\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:46.934040 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:46.933939    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:53:46.934040 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:46.934004    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv" podUID="3bc11d6a-c12d-4dab-8356-a8a05ee82f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:53:47.027923 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:47.027886    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85" event={"ID":"400327da-db74-4cce-be08-c75a73263374","Type":"ContainerDied","Data":"eed6093d22bd6a6c71c2b466ce455635a8b183b093b0f3ed9e42a9eab5932848"}
Oct 30 19:53:47.028120 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:47.027947    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eed6093d22bd6a6c71c2b466ce455635a8b183b093b0f3ed9e42a9eab5932848"
Oct 30 19:53:47.028120 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:47.028015    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"
Oct 30 19:53:47.667887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:47.667583    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:53:47.667887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:47.667645    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb" podUID="e473bfd6-107c-4cc1-a9f8-8f8e546b743b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 19:53:47.931840 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:47.931746    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:53:47.931980 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:47.931833    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv" podUID="3bc11d6a-c12d-4dab-8356-a8a05ee82f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:53:48.670884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:48.668167    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:53:48.670884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:48.668250    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb" podUID="e473bfd6-107c-4cc1-a9f8-8f8e546b743b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 19:53:48.970364 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:48.970270    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:48.971829 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:48.971805    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:48.972716 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:48.972693    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 19:53:49.672684 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:49.672650    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:49.673193 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:49.673089    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:49.673723 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:49.673708    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 19:53:55.306548 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.306513    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"]
Oct 30 19:53:55.307245 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.307226    4823 cpu_manager.go:414] "containerMap: removing container" podUID="2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f" containerName="revoke-keys"
Oct 30 19:53:55.307307 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.307247    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f" containerName="revoke-keys"
Oct 30 19:53:55.307307 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.307275    4823 cpu_manager.go:414] "containerMap: removing container" podUID="400327da-db74-4cce-be08-c75a73263374" containerName="revoke-keys"
Oct 30 19:53:55.307307 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.307281    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400327da-db74-4cce-be08-c75a73263374" containerName="revoke-keys"
Oct 30 19:53:55.307405 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.307382    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f" containerName="revoke-keys"
Oct 30 19:53:55.307405 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.307395    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="400327da-db74-4cce-be08-c75a73263374" containerName="revoke-keys"
Oct 30 19:53:55.311122 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.311105    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.334184 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.334141    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"]
Oct 30 19:53:55.471034 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.470998    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"]
Oct 30 19:53:55.476732 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.476664    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"
Oct 30 19:53:55.477312 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.477249    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"]
Oct 30 19:53:55.487294 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.487033    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:53:55.487410 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.487362    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad11ffea-d313-4369-81f8-70014b50603f-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.487578 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.487553    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.487641 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.487612    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.487700 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.487646    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.487750 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.487724    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad11ffea-d313-4369-81f8-70014b50603f-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.487800 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.487791    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.487855 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.487829    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.487969 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.487927    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad11ffea-d313-4369-81f8-70014b50603f-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.488021 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.488006    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkpkn\" (UniqueName: \"kubernetes.io/projected/ad11ffea-d313-4369-81f8-70014b50603f-kube-api-access-fkpkn\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.490087 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.490066    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:53:55.491563 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.491347    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:53:55.491563 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.491443    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:53:55.602092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.601615    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.602092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.601704    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad11ffea-d313-4369-81f8-70014b50603f-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.602092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.601799    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkpkn\" (UniqueName: \"kubernetes.io/projected/ad11ffea-d313-4369-81f8-70014b50603f-kube-api-access-fkpkn\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.602092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.601899    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4701206e-4vbjs\" (UID: \"dbdb0935-22a7-45e8-a4a2-a7053e726b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"
Oct 30 19:53:55.602092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.601942    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad11ffea-d313-4369-81f8-70014b50603f-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.602092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.602024    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slp2d\" (UniqueName: \"kubernetes.io/projected/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-kube-api-access-slp2d\") pod \"maas-api-revoke-sub-4701206e-4vbjs\" (UID: \"dbdb0935-22a7-45e8-a4a2-a7053e726b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"
Oct 30 19:53:55.602633 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.602112    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.602633 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.602146    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4701206e-4vbjs\" (UID: \"dbdb0935-22a7-45e8-a4a2-a7053e726b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"
Oct 30 19:53:55.602633 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.602194    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.602633 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.602231    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.602633 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.602310    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.602633 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.602324    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad11ffea-d313-4369-81f8-70014b50603f-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.602633 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.602378    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.603611 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.603530    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.603774 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.603627    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.604011 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.603990    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.604205 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.604178    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad11ffea-d313-4369-81f8-70014b50603f-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.605378 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.605353    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.606411 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.606384    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad11ffea-d313-4369-81f8-70014b50603f-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.616705 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.616678    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad11ffea-d313-4369-81f8-70014b50603f-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.617427 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.617343    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkpkn\" (UniqueName: \"kubernetes.io/projected/ad11ffea-d313-4369-81f8-70014b50603f-kube-api-access-fkpkn\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.624975 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.624947    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:55.705507 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.703455    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4701206e-4vbjs\" (UID: \"dbdb0935-22a7-45e8-a4a2-a7053e726b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"
Oct 30 19:53:55.705507 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.703566    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slp2d\" (UniqueName: \"kubernetes.io/projected/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-kube-api-access-slp2d\") pod \"maas-api-revoke-sub-4701206e-4vbjs\" (UID: \"dbdb0935-22a7-45e8-a4a2-a7053e726b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"
Oct 30 19:53:55.705507 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.703649    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4701206e-4vbjs\" (UID: \"dbdb0935-22a7-45e8-a4a2-a7053e726b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"
Oct 30 19:53:55.705507 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.704723    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4701206e-4vbjs\" (UID: \"dbdb0935-22a7-45e8-a4a2-a7053e726b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"
Oct 30 19:53:55.715241 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.715214    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4701206e-4vbjs\" (UID: \"dbdb0935-22a7-45e8-a4a2-a7053e726b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"
Oct 30 19:53:55.720647 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.720621    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slp2d\" (UniqueName: \"kubernetes.io/projected/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-kube-api-access-slp2d\") pod \"maas-api-revoke-sub-4701206e-4vbjs\" (UID: \"dbdb0935-22a7-45e8-a4a2-a7053e726b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"
Oct 30 19:53:55.793452 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.793421    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"
Oct 30 19:53:55.939115 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:53:55.939073    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad11ffea_d313_4369_81f8_70014b50603f.slice/crio-55d4f6c0d4f597a9890dfa7107369ab3be5590a7375f0a38f3a3561a6195bdd7 WatchSource:0}: Error finding container 55d4f6c0d4f597a9890dfa7107369ab3be5590a7375f0a38f3a3561a6195bdd7: Status 404 returned error can't find the container with id 55d4f6c0d4f597a9890dfa7107369ab3be5590a7375f0a38f3a3561a6195bdd7
Oct 30 19:53:55.939292 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.939265    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"]
Oct 30 19:53:55.942313 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.942272    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:55.942379 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.942360    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:55.942423 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:55.942403    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:56.006056 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.006032    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"]
Oct 30 19:53:56.011458 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.011441    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"
Oct 30 19:53:56.031102 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.031075    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"]
Oct 30 19:53:56.043501 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.043469    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"]
Oct 30 19:53:56.071572 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.071543    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.096803 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.096770    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"]
Oct 30 19:53:56.114179 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.114132    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.116584 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.116556    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.125997 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.125975    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48611f0a-b2a4-4499-a80e-5d2ef398981a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29e7fc77-xj7cz\" (UID: \"48611f0a-b2a4-4499-a80e-5d2ef398981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"
Oct 30 19:53:56.126210 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.126188    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.127245 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.116408    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"]
Oct 30 19:53:56.127404 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.127382    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.127556 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.127538    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.127692 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.127674    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48611f0a-b2a4-4499-a80e-5d2ef398981a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29e7fc77-xj7cz\" (UID: \"48611f0a-b2a4-4499-a80e-5d2ef398981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"
Oct 30 19:53:56.127928 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.127907    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfn8l\" (UniqueName: \"kubernetes.io/projected/48611f0a-b2a4-4499-a80e-5d2ef398981a-kube-api-access-jfn8l\") pod \"maas-api-revoke-sub-29e7fc77-xj7cz\" (UID: \"48611f0a-b2a4-4499-a80e-5d2ef398981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"
Oct 30 19:53:56.128133 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.128112    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hq6c\" (UniqueName: \"kubernetes.io/projected/5839d58f-f3c6-4adb-a291-9f33b1659f9a-kube-api-access-6hq6c\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.128321 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.128301    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.128475 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.128457    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.128617 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.128597    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.135734 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.135712    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm" event={"ID":"ad11ffea-d313-4369-81f8-70014b50603f","Type":"ContainerStarted","Data":"f870e6a685e7ce309cabb73665dfaddc0de0e9e39ad63245a98ccb6100e17358"}
Oct 30 19:53:56.135951 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.135913    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm" event={"ID":"ad11ffea-d313-4369-81f8-70014b50603f","Type":"ContainerStarted","Data":"55d4f6c0d4f597a9890dfa7107369ab3be5590a7375f0a38f3a3561a6195bdd7"}
Oct 30 19:53:56.212973 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.212924    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm" podStartSLOduration=1.2129096129999999 podStartE2EDuration="1.212909613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:56.207992231 +0000 UTC m=+1436.100606556" watchObservedRunningTime="2026-10-30 19:53:56.212909613 +0000 UTC m=+1436.105523938"
Oct 30 19:53:56.230110 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.230064    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfn8l\" (UniqueName: \"kubernetes.io/projected/48611f0a-b2a4-4499-a80e-5d2ef398981a-kube-api-access-jfn8l\") pod \"maas-api-revoke-sub-29e7fc77-xj7cz\" (UID: \"48611f0a-b2a4-4499-a80e-5d2ef398981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"
Oct 30 19:53:56.230110 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.230143    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hq6c\" (UniqueName: \"kubernetes.io/projected/5839d58f-f3c6-4adb-a291-9f33b1659f9a-kube-api-access-6hq6c\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.230763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.230214    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.230763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.230265    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.230763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.230298    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.230763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.230353    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.230763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.230447    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.230763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.230531    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48611f0a-b2a4-4499-a80e-5d2ef398981a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29e7fc77-xj7cz\" (UID: \"48611f0a-b2a4-4499-a80e-5d2ef398981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"
Oct 30 19:53:56.230763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.230562    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.230763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.230638    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.230763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.230687    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.230763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.230714    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48611f0a-b2a4-4499-a80e-5d2ef398981a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29e7fc77-xj7cz\" (UID: \"48611f0a-b2a4-4499-a80e-5d2ef398981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"
Oct 30 19:53:56.230763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.230758    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.231886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.231153    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.231886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.231194    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.231886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.231250    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.231886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.231353    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.231886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.231837    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48611f0a-b2a4-4499-a80e-5d2ef398981a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29e7fc77-xj7cz\" (UID: \"48611f0a-b2a4-4499-a80e-5d2ef398981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"
Oct 30 19:53:56.234039 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.234018    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48611f0a-b2a4-4499-a80e-5d2ef398981a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29e7fc77-xj7cz\" (UID: \"48611f0a-b2a4-4499-a80e-5d2ef398981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"
Oct 30 19:53:56.234315 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.234291    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.234434 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.234413    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.255301 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.255271    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.263543 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.263521    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hq6c\" (UniqueName: \"kubernetes.io/projected/5839d58f-f3c6-4adb-a291-9f33b1659f9a-kube-api-access-6hq6c\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.264007 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.263987    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfn8l\" (UniqueName: \"kubernetes.io/projected/48611f0a-b2a4-4499-a80e-5d2ef398981a-kube-api-access-jfn8l\") pod \"maas-api-revoke-sub-29e7fc77-xj7cz\" (UID: \"48611f0a-b2a4-4499-a80e-5d2ef398981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"
Oct 30 19:53:56.328135 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.328100    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"
Oct 30 19:53:56.394838 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.394813    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:56.625639 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.625599    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:53:56.626054 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:53:56.626027    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48611f0a_b2a4_4499_a80e_5d2ef398981a.slice/crio-ec3e05b302bfe4c52afa6519dfa9d770c0da67c0a43825c5e28ca18443dd9079 WatchSource:0}: Error finding container ec3e05b302bfe4c52afa6519dfa9d770c0da67c0a43825c5e28ca18443dd9079: Status 404 returned error can't find the container with id ec3e05b302bfe4c52afa6519dfa9d770c0da67c0a43825c5e28ca18443dd9079
Oct 30 19:53:56.626286 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.626267    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"]
Oct 30 19:53:56.727125 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.727067    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"]
Oct 30 19:53:56.727871 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:53:56.727831    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5839d58f_f3c6_4adb_a291_9f33b1659f9a.slice/crio-35897b5987093446425af04b859fb064127cc1a972f0ad96fd5caab1f58a2067 WatchSource:0}: Error finding container 35897b5987093446425af04b859fb064127cc1a972f0ad96fd5caab1f58a2067: Status 404 returned error can't find the container with id 35897b5987093446425af04b859fb064127cc1a972f0ad96fd5caab1f58a2067
Oct 30 19:53:56.733122 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.733075    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:56.733233 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.733171    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:56.733233 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:56.733219    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:53:57.146793 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:57.146698    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs" event={"ID":"dbdb0935-22a7-45e8-a4a2-a7053e726b8b","Type":"ContainerStarted","Data":"0596e9013732de95eaee0ac5011a53be7813f26c950d349cccf81f23ff93d141"}
Oct 30 19:53:57.146793 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:57.146763    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs" event={"ID":"dbdb0935-22a7-45e8-a4a2-a7053e726b8b","Type":"ContainerStarted","Data":"afce2d6d92efb19c1937440a93cfd96fed7e627c5c0576e21b0a6eae56f98412"}
Oct 30 19:53:57.149474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:57.149447    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m" event={"ID":"5839d58f-f3c6-4adb-a291-9f33b1659f9a","Type":"ContainerStarted","Data":"ea04bdfed1191bbc27a516123c3045133ae9e461716e5412545bd139f9eddf68"}
Oct 30 19:53:57.149588 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:57.149490    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m" event={"ID":"5839d58f-f3c6-4adb-a291-9f33b1659f9a","Type":"ContainerStarted","Data":"35897b5987093446425af04b859fb064127cc1a972f0ad96fd5caab1f58a2067"}
Oct 30 19:53:57.151213 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:57.151189    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz" event={"ID":"48611f0a-b2a4-4499-a80e-5d2ef398981a","Type":"ContainerStarted","Data":"ec3e05b302bfe4c52afa6519dfa9d770c0da67c0a43825c5e28ca18443dd9079"}
Oct 30 19:53:57.188177 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:57.188107    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs" podStartSLOduration=2.188086606 podStartE2EDuration="2.188086606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:57.182413913 +0000 UTC m=+1437.075028237" watchObservedRunningTime="2026-10-30 19:53:57.188086606 +0000 UTC m=+1437.080700931"
Oct 30 19:53:57.213914 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:57.213849    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m" podStartSLOduration=1.213835065 podStartE2EDuration="1.213835065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:57.208242972 +0000 UTC m=+1437.100857295" watchObservedRunningTime="2026-10-30 19:53:57.213835065 +0000 UTC m=+1437.106449478"
Oct 30 19:53:57.395944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:57.395903    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:53:57.397593 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:57.397527    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:53:57.397593 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:57.397578    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m" podUID="5839d58f-f3c6-4adb-a291-9f33b1659f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 19:53:57.627157 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:57.627126    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:53:57.627305 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:57.627182    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm" podUID="ad11ffea-d313-4369-81f8-70014b50603f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:53:58.211194 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:58.211156    4823 generic.go:356] "Generic (PLEG): container finished" podID="48611f0a-b2a4-4499-a80e-5d2ef398981a" containerID="2699cd3dde34a6e745bad857603f64cc289e3f88c9c01ec4f518af99e514aacd" exitCode=0
Oct 30 19:53:58.211486 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:58.211304    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz" event={"ID":"48611f0a-b2a4-4499-a80e-5d2ef398981a","Type":"ContainerDied","Data":"2699cd3dde34a6e745bad857603f64cc289e3f88c9c01ec4f518af99e514aacd"}
Oct 30 19:53:58.213571 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:58.213545    4823 generic.go:356] "Generic (PLEG): container finished" podID="dbdb0935-22a7-45e8-a4a2-a7053e726b8b" containerID="0596e9013732de95eaee0ac5011a53be7813f26c950d349cccf81f23ff93d141" exitCode=0
Oct 30 19:53:58.213937 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:58.213914    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs" event={"ID":"dbdb0935-22a7-45e8-a4a2-a7053e726b8b","Type":"ContainerDied","Data":"0596e9013732de95eaee0ac5011a53be7813f26c950d349cccf81f23ff93d141"}
Oct 30 19:53:58.399884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:58.399354    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:53:58.399884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:58.399411    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m" podUID="5839d58f-f3c6-4adb-a291-9f33b1659f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 19:53:58.626110 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:58.626081    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:53:58.626361 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:58.626327    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm" podUID="ad11ffea-d313-4369-81f8-70014b50603f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:53:59.408885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.408339    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:53:59.408885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.408401    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m" podUID="5839d58f-f3c6-4adb-a291-9f33b1659f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 19:53:59.608960 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.589056    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"
Oct 30 19:53:59.626190 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.625992    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:53:59.626190 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.626047    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm" podUID="ad11ffea-d313-4369-81f8-70014b50603f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:53:59.710833 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.710615    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"
Oct 30 19:53:59.736693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.735898    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48611f0a-b2a4-4499-a80e-5d2ef398981a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48611f0a-b2a4-4499-a80e-5d2ef398981a-maas-api-xks-ca\") pod \"48611f0a-b2a4-4499-a80e-5d2ef398981a\" (UID: \"48611f0a-b2a4-4499-a80e-5d2ef398981a\") "
Oct 30 19:53:59.736693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.735942    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48611f0a-b2a4-4499-a80e-5d2ef398981a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48611f0a-b2a4-4499-a80e-5d2ef398981a-maas-api-openshift-ca\") pod \"48611f0a-b2a4-4499-a80e-5d2ef398981a\" (UID: \"48611f0a-b2a4-4499-a80e-5d2ef398981a\") "
Oct 30 19:53:59.736693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.736028    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48611f0a-b2a4-4499-a80e-5d2ef398981a-kube-api-access-jfn8l\" (UniqueName: \"kubernetes.io/projected/48611f0a-b2a4-4499-a80e-5d2ef398981a-kube-api-access-jfn8l\") pod \"48611f0a-b2a4-4499-a80e-5d2ef398981a\" (UID: \"48611f0a-b2a4-4499-a80e-5d2ef398981a\") "
Oct 30 19:53:59.737268 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.737236    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48611f0a-b2a4-4499-a80e-5d2ef398981a-maas-api-openshift-ca" pod "48611f0a-b2a4-4499-a80e-5d2ef398981a" (UID: "48611f0a-b2a4-4499-a80e-5d2ef398981a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:59.738650 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.738617    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48611f0a-b2a4-4499-a80e-5d2ef398981a-kube-api-access-jfn8l" pod "48611f0a-b2a4-4499-a80e-5d2ef398981a" (UID: "48611f0a-b2a4-4499-a80e-5d2ef398981a"). InnerVolumeSpecName "kube-api-access-jfn8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:59.738952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.738932    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48611f0a-b2a4-4499-a80e-5d2ef398981a-maas-api-xks-ca" pod "48611f0a-b2a4-4499-a80e-5d2ef398981a" (UID: "48611f0a-b2a4-4499-a80e-5d2ef398981a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:59.837357 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.837266    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-maas-api-xks-ca\") pod \"dbdb0935-22a7-45e8-a4a2-a7053e726b8b\" (UID: \"dbdb0935-22a7-45e8-a4a2-a7053e726b8b\") "
Oct 30 19:53:59.837511 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.837466    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-kube-api-access-slp2d\" (UniqueName: \"kubernetes.io/projected/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-kube-api-access-slp2d\") pod \"dbdb0935-22a7-45e8-a4a2-a7053e726b8b\" (UID: \"dbdb0935-22a7-45e8-a4a2-a7053e726b8b\") "
Oct 30 19:53:59.837567 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.837531    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-maas-api-openshift-ca\") pod \"dbdb0935-22a7-45e8-a4a2-a7053e726b8b\" (UID: \"dbdb0935-22a7-45e8-a4a2-a7053e726b8b\") "
Oct 30 19:53:59.838205 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.838165    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-maas-api-openshift-ca" pod "dbdb0935-22a7-45e8-a4a2-a7053e726b8b" (UID: "dbdb0935-22a7-45e8-a4a2-a7053e726b8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:59.838685 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.838663    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfn8l\" (UniqueName: \"kubernetes.io/projected/48611f0a-b2a4-4499-a80e-5d2ef398981a-kube-api-access-jfn8l\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:59.838818 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.838803    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:59.838943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.838930    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48611f0a-b2a4-4499-a80e-5d2ef398981a-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:59.839030 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.839018    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48611f0a-b2a4-4499-a80e-5d2ef398981a-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:59.839781 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.839752    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-maas-api-xks-ca" pod "dbdb0935-22a7-45e8-a4a2-a7053e726b8b" (UID: "dbdb0935-22a7-45e8-a4a2-a7053e726b8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:59.840390 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.840374    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-kube-api-access-slp2d" pod "dbdb0935-22a7-45e8-a4a2-a7053e726b8b" (UID: "dbdb0935-22a7-45e8-a4a2-a7053e726b8b"). InnerVolumeSpecName "kube-api-access-slp2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:59.940832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.940798    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:59.940832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:53:59.940828    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slp2d\" (UniqueName: \"kubernetes.io/projected/dbdb0935-22a7-45e8-a4a2-a7053e726b8b-kube-api-access-slp2d\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:00.246798 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:00.246768    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"
Oct 30 19:54:00.247070 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:00.246786    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz" event={"ID":"48611f0a-b2a4-4499-a80e-5d2ef398981a","Type":"ContainerDied","Data":"ec3e05b302bfe4c52afa6519dfa9d770c0da67c0a43825c5e28ca18443dd9079"}
Oct 30 19:54:00.247153 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:00.247135    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec3e05b302bfe4c52afa6519dfa9d770c0da67c0a43825c5e28ca18443dd9079"
Oct 30 19:54:00.248552 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:00.248528    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs" event={"ID":"dbdb0935-22a7-45e8-a4a2-a7053e726b8b","Type":"ContainerDied","Data":"afce2d6d92efb19c1937440a93cfd96fed7e627c5c0576e21b0a6eae56f98412"}
Oct 30 19:54:00.248666 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:00.248573    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afce2d6d92efb19c1937440a93cfd96fed7e627c5c0576e21b0a6eae56f98412"
Oct 30 19:54:00.248666 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:00.248578    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"
Oct 30 19:54:00.402115 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:00.402080    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:54:00.402400 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:00.402368    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m" podUID="5839d58f-f3c6-4adb-a291-9f33b1659f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 19:54:00.625753 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:00.625717    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:54:00.626239 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:00.625788    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm" podUID="ad11ffea-d313-4369-81f8-70014b50603f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:54:01.403213 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:01.403171    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:54:01.403531 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:01.403501    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m" podUID="5839d58f-f3c6-4adb-a291-9f33b1659f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 19:54:01.625480 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:01.625393    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:54:01.625480 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:01.625449    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm" podUID="ad11ffea-d313-4369-81f8-70014b50603f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:54:02.047197 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:54:02.047106    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:54:02.395870 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:02.395829    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:54:02.396027 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:02.395897    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m" podUID="5839d58f-f3c6-4adb-a291-9f33b1659f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 19:54:02.626291 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:02.626251    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:54:02.626486 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:02.626344    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm" podUID="ad11ffea-d313-4369-81f8-70014b50603f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:54:03.398217 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:03.398176    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:54:03.398973 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:03.398932    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m" podUID="5839d58f-f3c6-4adb-a291-9f33b1659f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 19:54:03.838936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:03.838824    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:54:03.839444 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:03.839421    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:54:03.840386 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:03.840354    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 19:54:04.851293 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:04.851242    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:54:04.853224 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:04.853136    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:54:04.853973 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:04.853937    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 19:54:05.849533 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:05.849499    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"]
Oct 30 19:54:05.850307 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:05.850285    4823 cpu_manager.go:414] "containerMap: removing container" podUID="dbdb0935-22a7-45e8-a4a2-a7053e726b8b" containerName="revoke-keys"
Oct 30 19:54:05.850307 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:05.850306    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbdb0935-22a7-45e8-a4a2-a7053e726b8b" containerName="revoke-keys"
Oct 30 19:54:05.850482 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:05.850324    4823 cpu_manager.go:414] "containerMap: removing container" podUID="48611f0a-b2a4-4499-a80e-5d2ef398981a" containerName="revoke-keys"
Oct 30 19:54:05.850482 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:05.850331    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48611f0a-b2a4-4499-a80e-5d2ef398981a" containerName="revoke-keys"
Oct 30 19:54:05.850579 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:05.850525    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbdb0935-22a7-45e8-a4a2-a7053e726b8b" containerName="revoke-keys"
Oct 30 19:54:05.850579 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:05.850544    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="48611f0a-b2a4-4499-a80e-5d2ef398981a" containerName="revoke-keys"
Oct 30 19:54:05.855828 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:05.855808    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"
Oct 30 19:54:05.878763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:05.878738    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:54:05.886000 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:05.883087    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:54:05.886000 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:05.883208    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:54:05.886000 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:05.883279    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:54:05.886000 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:05.884512    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"]
Oct 30 19:54:05.938892 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:05.936499    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe5fa903-ssk7x\" (UID: \"e0b302e4-25dd-4ef6-814b-27075b1c1ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"
Oct 30 19:54:05.938892 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:05.936556    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe5fa903-ssk7x\" (UID: \"e0b302e4-25dd-4ef6-814b-27075b1c1ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"
Oct 30 19:54:05.938892 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:05.936715    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrck6\" (UniqueName: \"kubernetes.io/projected/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-kube-api-access-rrck6\") pod \"maas-api-revoke-sub-fe5fa903-ssk7x\" (UID: \"e0b302e4-25dd-4ef6-814b-27075b1c1ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"
Oct 30 19:54:06.047891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:06.046945    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrck6\" (UniqueName: \"kubernetes.io/projected/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-kube-api-access-rrck6\") pod \"maas-api-revoke-sub-fe5fa903-ssk7x\" (UID: \"e0b302e4-25dd-4ef6-814b-27075b1c1ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"
Oct 30 19:54:06.047891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:06.047216    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe5fa903-ssk7x\" (UID: \"e0b302e4-25dd-4ef6-814b-27075b1c1ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"
Oct 30 19:54:06.047891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:06.047250    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe5fa903-ssk7x\" (UID: \"e0b302e4-25dd-4ef6-814b-27075b1c1ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"
Oct 30 19:54:06.101937 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:06.100877    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe5fa903-ssk7x\" (UID: \"e0b302e4-25dd-4ef6-814b-27075b1c1ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"
Oct 30 19:54:06.142411 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:06.132829    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrck6\" (UniqueName: \"kubernetes.io/projected/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-kube-api-access-rrck6\") pod \"maas-api-revoke-sub-fe5fa903-ssk7x\" (UID: \"e0b302e4-25dd-4ef6-814b-27075b1c1ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"
Oct 30 19:54:06.142411 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:06.135442    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe5fa903-ssk7x\" (UID: \"e0b302e4-25dd-4ef6-814b-27075b1c1ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"
Oct 30 19:54:06.170107 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:06.170075    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"
Oct 30 19:54:06.542036 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:54:06.541985    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b302e4_25dd_4ef6_814b_27075b1c1ca3.slice/crio-ed53b5314ba55c2f5bf4696b66e5cfd8885aa3cdc1d585f1501c4b3c795c6b23 WatchSource:0}: Error finding container ed53b5314ba55c2f5bf4696b66e5cfd8885aa3cdc1d585f1501c4b3c795c6b23: Status 404 returned error can't find the container with id ed53b5314ba55c2f5bf4696b66e5cfd8885aa3cdc1d585f1501c4b3c795c6b23
Oct 30 19:54:06.574886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:06.555112    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"]
Oct 30 19:54:07.492887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:07.481551    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x" event={"ID":"e0b302e4-25dd-4ef6-814b-27075b1c1ca3","Type":"ContainerStarted","Data":"cd3873a36219c74a78b62a5e2fba580ea853d764848b410c259ad35d27451823"}
Oct 30 19:54:07.492887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:07.481633    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x" event={"ID":"e0b302e4-25dd-4ef6-814b-27075b1c1ca3","Type":"ContainerStarted","Data":"ed53b5314ba55c2f5bf4696b66e5cfd8885aa3cdc1d585f1501c4b3c795c6b23"}
Oct 30 19:54:07.532978 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:07.528451    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x" podStartSLOduration=2.528428361 podStartE2EDuration="2.528428361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:07.517029304 +0000 UTC m=+1447.409643633" watchObservedRunningTime="2026-10-30 19:54:07.528428361 +0000 UTC m=+1447.421042684"
Oct 30 19:54:08.181947 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:08.179046    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.198889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:08.196184    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.220135 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:08.220087    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.495956 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:08.492948    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.495956 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:08.493312    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.558894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:08.549042    4823 generic.go:356] "Generic (PLEG): container finished" podID="e0b302e4-25dd-4ef6-814b-27075b1c1ca3" containerID="cd3873a36219c74a78b62a5e2fba580ea853d764848b410c259ad35d27451823" exitCode=0
Oct 30 19:54:08.558894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:08.549166    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x" event={"ID":"e0b302e4-25dd-4ef6-814b-27075b1c1ca3","Type":"ContainerDied","Data":"cd3873a36219c74a78b62a5e2fba580ea853d764848b410c259ad35d27451823"}
Oct 30 19:54:08.665017 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:08.664968    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.665611 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:08.665578    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:54:08.763144 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:08.763019    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:08.805014 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:08.804966    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:54:08.852095 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:08.852023    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.057939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.055985    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.198887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.197963    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.198887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.198783    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:54:09.396899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.395761    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.396899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.396599    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:54:09.552887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.498996    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.552887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.499310    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:54:09.552887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.499554    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:54:09.552887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.499625    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:54:09.633565 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.633522    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.666406 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.666301    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.667926 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.667886    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.669223 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.669202    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:54:09.669472 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.669455    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:54:09.817896 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.741649    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.817896 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.776880    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:09.817896 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.792913    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:54:09.825728 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:09.825678    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:10.140508 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:10.098009    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:54:10.140508 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:10.099071    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:54:10.140508 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:10.112010    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:54:10.232887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:10.205721    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:10.396238 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:10.396194    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:10.561783 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:10.561728    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:54:10.622887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:10.562934    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:54:10.681992 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:10.673078    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:10.777399 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:10.713877    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:10.778712 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:10.777174    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:54:11.098732 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.098621    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:11.137501 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.137447    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:11.261888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.245329    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:11.497745 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.481550    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:54:11.566379 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.566333    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:54:11.604020 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.603962    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:11.631991 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.613023    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:54:11.811885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.793143    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:54:11.820178 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.820129    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:11.952614 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.943001    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:54:11.957362 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.957334    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:54:11.960650 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.959648    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:54:11.962438 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.962419    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:54:11.962605 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.962594    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:54:11.962979 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.962963    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:54:11.963685 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.963656    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:11.982496 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.981570    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:54:11.987213 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.987186    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"
Oct 30 19:54:11.992093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:11.992068    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:54:12.080087 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.080008    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"
Oct 30 19:54:12.080226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.080087    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x" event={"ID":"e0b302e4-25dd-4ef6-814b-27075b1c1ca3","Type":"ContainerDied","Data":"ed53b5314ba55c2f5bf4696b66e5cfd8885aa3cdc1d585f1501c4b3c795c6b23"}
Oct 30 19:54:12.080226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.080141    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed53b5314ba55c2f5bf4696b66e5cfd8885aa3cdc1d585f1501c4b3c795c6b23"
Oct 30 19:54:12.161623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.161588    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-kube-api-access-rrck6\" (UniqueName: \"kubernetes.io/projected/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-kube-api-access-rrck6\") pod \"e0b302e4-25dd-4ef6-814b-27075b1c1ca3\" (UID: \"e0b302e4-25dd-4ef6-814b-27075b1c1ca3\") "
Oct 30 19:54:12.162092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.162047    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-maas-api-openshift-ca\") pod \"e0b302e4-25dd-4ef6-814b-27075b1c1ca3\" (UID: \"e0b302e4-25dd-4ef6-814b-27075b1c1ca3\") "
Oct 30 19:54:12.162231 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.162173    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-maas-api-xks-ca\") pod \"e0b302e4-25dd-4ef6-814b-27075b1c1ca3\" (UID: \"e0b302e4-25dd-4ef6-814b-27075b1c1ca3\") "
Oct 30 19:54:12.167218 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.167190    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-maas-api-openshift-ca" pod "e0b302e4-25dd-4ef6-814b-27075b1c1ca3" (UID: "e0b302e4-25dd-4ef6-814b-27075b1c1ca3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:12.167609 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.167582    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-kube-api-access-rrck6" pod "e0b302e4-25dd-4ef6-814b-27075b1c1ca3" (UID: "e0b302e4-25dd-4ef6-814b-27075b1c1ca3"). InnerVolumeSpecName "kube-api-access-rrck6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:12.171689 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.171663    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-maas-api-xks-ca" pod "e0b302e4-25dd-4ef6-814b-27075b1c1ca3" (UID: "e0b302e4-25dd-4ef6-814b-27075b1c1ca3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:12.264776 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.264743    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrck6\" (UniqueName: \"kubernetes.io/projected/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-kube-api-access-rrck6\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:12.264776 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.264775    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:12.265037 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.264789    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b302e4-25dd-4ef6-814b-27075b1c1ca3-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:12.807332 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.807300    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"]
Oct 30 19:54:12.808011 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.807986    4823 cpu_manager.go:414] "containerMap: removing container" podUID="e0b302e4-25dd-4ef6-814b-27075b1c1ca3" containerName="revoke-keys"
Oct 30 19:54:12.808011 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.808001    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b302e4-25dd-4ef6-814b-27075b1c1ca3" containerName="revoke-keys"
Oct 30 19:54:12.808162 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.808152    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b302e4-25dd-4ef6-814b-27075b1c1ca3" containerName="revoke-keys"
Oct 30 19:54:12.812153 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.812129    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"
Oct 30 19:54:12.833409 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.833096    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:54:12.833409 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.833113    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:54:12.833409 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.833188    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:54:12.833702 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.833435    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:54:12.842000 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.841973    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"]
Oct 30 19:54:12.873876 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.873825    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34658db-8f32-48b4-a690-ff7fac03644c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d55ccd1-v9gl5\" (UID: \"e34658db-8f32-48b4-a690-ff7fac03644c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"
Oct 30 19:54:12.874069 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.873914    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34658db-8f32-48b4-a690-ff7fac03644c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d55ccd1-v9gl5\" (UID: \"e34658db-8f32-48b4-a690-ff7fac03644c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"
Oct 30 19:54:12.874069 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.874046    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9z4w\" (UniqueName: \"kubernetes.io/projected/e34658db-8f32-48b4-a690-ff7fac03644c-kube-api-access-x9z4w\") pod \"maas-api-revoke-sub-4d55ccd1-v9gl5\" (UID: \"e34658db-8f32-48b4-a690-ff7fac03644c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"
Oct 30 19:54:12.975022 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.974985    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9z4w\" (UniqueName: \"kubernetes.io/projected/e34658db-8f32-48b4-a690-ff7fac03644c-kube-api-access-x9z4w\") pod \"maas-api-revoke-sub-4d55ccd1-v9gl5\" (UID: \"e34658db-8f32-48b4-a690-ff7fac03644c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"
Oct 30 19:54:12.975173 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.975105    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34658db-8f32-48b4-a690-ff7fac03644c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d55ccd1-v9gl5\" (UID: \"e34658db-8f32-48b4-a690-ff7fac03644c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"
Oct 30 19:54:12.975250 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.975170    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34658db-8f32-48b4-a690-ff7fac03644c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d55ccd1-v9gl5\" (UID: \"e34658db-8f32-48b4-a690-ff7fac03644c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"
Oct 30 19:54:12.975849 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.975819    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34658db-8f32-48b4-a690-ff7fac03644c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d55ccd1-v9gl5\" (UID: \"e34658db-8f32-48b4-a690-ff7fac03644c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"
Oct 30 19:54:12.977608 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.977581    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34658db-8f32-48b4-a690-ff7fac03644c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d55ccd1-v9gl5\" (UID: \"e34658db-8f32-48b4-a690-ff7fac03644c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"
Oct 30 19:54:12.992511 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:12.992485    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9z4w\" (UniqueName: \"kubernetes.io/projected/e34658db-8f32-48b4-a690-ff7fac03644c-kube-api-access-x9z4w\") pod \"maas-api-revoke-sub-4d55ccd1-v9gl5\" (UID: \"e34658db-8f32-48b4-a690-ff7fac03644c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"
Oct 30 19:54:13.029985 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:13.029953    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"]
Oct 30 19:54:13.034715 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:13.034694    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"
Oct 30 19:54:13.045823 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:13.045801    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"]
Oct 30 19:54:13.076797 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:13.076712    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrwt7\" (UniqueName: \"kubernetes.io/projected/d37f565f-b457-42f3-9854-5f94bc8d60f2-kube-api-access-zrwt7\") pod \"maas-api-revoke-sub-f8556e06-zpr2k\" (UID: \"d37f565f-b457-42f3-9854-5f94bc8d60f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"
Oct 30 19:54:13.076977 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:13.076855    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37f565f-b457-42f3-9854-5f94bc8d60f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8556e06-zpr2k\" (UID: \"d37f565f-b457-42f3-9854-5f94bc8d60f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"
Oct 30 19:54:13.077076 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:13.077046    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37f565f-b457-42f3-9854-5f94bc8d60f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8556e06-zpr2k\" (UID: \"d37f565f-b457-42f3-9854-5f94bc8d60f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"
Oct 30 19:54:13.131172 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:13.131134    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"
Oct 30 19:54:13.178213 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:13.178174    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37f565f-b457-42f3-9854-5f94bc8d60f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8556e06-zpr2k\" (UID: \"d37f565f-b457-42f3-9854-5f94bc8d60f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"
Oct 30 19:54:13.178428 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:13.178408    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrwt7\" (UniqueName: \"kubernetes.io/projected/d37f565f-b457-42f3-9854-5f94bc8d60f2-kube-api-access-zrwt7\") pod \"maas-api-revoke-sub-f8556e06-zpr2k\" (UID: \"d37f565f-b457-42f3-9854-5f94bc8d60f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"
Oct 30 19:54:13.178514 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:13.178500    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37f565f-b457-42f3-9854-5f94bc8d60f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8556e06-zpr2k\" (UID: \"d37f565f-b457-42f3-9854-5f94bc8d60f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"
Oct 30 19:54:13.179331 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:13.179300    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37f565f-b457-42f3-9854-5f94bc8d60f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8556e06-zpr2k\" (UID: \"d37f565f-b457-42f3-9854-5f94bc8d60f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"
Oct 30 19:54:13.180599 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:13.180568    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37f565f-b457-42f3-9854-5f94bc8d60f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8556e06-zpr2k\" (UID: \"d37f565f-b457-42f3-9854-5f94bc8d60f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"
Oct 30 19:54:13.198653 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:13.198620    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrwt7\" (UniqueName: \"kubernetes.io/projected/d37f565f-b457-42f3-9854-5f94bc8d60f2-kube-api-access-zrwt7\") pod \"maas-api-revoke-sub-f8556e06-zpr2k\" (UID: \"d37f565f-b457-42f3-9854-5f94bc8d60f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"
Oct 30 19:54:13.353988 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:13.353894    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"
Oct 30 19:54:13.437357 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:13.437047    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"]
Oct 30 19:54:13.442054 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:54:13.442014    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34658db_8f32_48b4_a690_ff7fac03644c.slice/crio-0e07a0722d8e22760e5245425f1f9af379e91f14ee212295a52aa8b54efdad43 WatchSource:0}: Error finding container 0e07a0722d8e22760e5245425f1f9af379e91f14ee212295a52aa8b54efdad43: Status 404 returned error can't find the container with id 0e07a0722d8e22760e5245425f1f9af379e91f14ee212295a52aa8b54efdad43
Oct 30 19:54:13.669987 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:13.669923    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"]
Oct 30 19:54:14.028010 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.027973    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"]
Oct 30 19:54:14.032769 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.032673    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"
Oct 30 19:54:14.044712 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.044685    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"]
Oct 30 19:54:14.099081 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.099047    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5" event={"ID":"e34658db-8f32-48b4-a690-ff7fac03644c","Type":"ContainerStarted","Data":"d63b7fddd7578d23f6fafbd211d8d6e56b4fb0460dd1504ba0440ac237a9d21f"}
Oct 30 19:54:14.099260 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.099109    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5" event={"ID":"e34658db-8f32-48b4-a690-ff7fac03644c","Type":"ContainerStarted","Data":"0e07a0722d8e22760e5245425f1f9af379e91f14ee212295a52aa8b54efdad43"}
Oct 30 19:54:14.100766 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.100741    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k" event={"ID":"d37f565f-b457-42f3-9854-5f94bc8d60f2","Type":"ContainerStarted","Data":"bb815c521bf1aea059dc6cc4dbb99aeac9ab3000b12861f5b14035818e082aec"}
Oct 30 19:54:14.100934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.100783    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k" event={"ID":"d37f565f-b457-42f3-9854-5f94bc8d60f2","Type":"ContainerStarted","Data":"752cc9c01e28a394fee4968ee834a62b260288efc45d41e4c3f48f3195cde580"}
Oct 30 19:54:14.110490 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.110461    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqwgd\" (UniqueName: \"kubernetes.io/projected/4e7b0082-74b4-43f3-bf68-d67397801800-kube-api-access-zqwgd\") pod \"maas-api-revoke-sub-70893d69-zhqzc\" (UID: \"4e7b0082-74b4-43f3-bf68-d67397801800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"
Oct 30 19:54:14.110604 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.110582    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e7b0082-74b4-43f3-bf68-d67397801800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70893d69-zhqzc\" (UID: \"4e7b0082-74b4-43f3-bf68-d67397801800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"
Oct 30 19:54:14.110649 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.110621    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e7b0082-74b4-43f3-bf68-d67397801800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70893d69-zhqzc\" (UID: \"4e7b0082-74b4-43f3-bf68-d67397801800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"
Oct 30 19:54:14.121181 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.121122    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5" podStartSLOduration=2.121111033 podStartE2EDuration="2.121111033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:14.118410332 +0000 UTC m=+1454.011024657" watchObservedRunningTime="2026-10-30 19:54:14.121111033 +0000 UTC m=+1454.013725388"
Oct 30 19:54:14.139666 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.139619    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k" podStartSLOduration=2.139607397 podStartE2EDuration="2.139607397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:14.136988491 +0000 UTC m=+1454.029602818" watchObservedRunningTime="2026-10-30 19:54:14.139607397 +0000 UTC m=+1454.032221740"
Oct 30 19:54:14.212174 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.212075    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqwgd\" (UniqueName: \"kubernetes.io/projected/4e7b0082-74b4-43f3-bf68-d67397801800-kube-api-access-zqwgd\") pod \"maas-api-revoke-sub-70893d69-zhqzc\" (UID: \"4e7b0082-74b4-43f3-bf68-d67397801800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"
Oct 30 19:54:14.212330 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.212176    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e7b0082-74b4-43f3-bf68-d67397801800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70893d69-zhqzc\" (UID: \"4e7b0082-74b4-43f3-bf68-d67397801800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"
Oct 30 19:54:14.212330 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.212201    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e7b0082-74b4-43f3-bf68-d67397801800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70893d69-zhqzc\" (UID: \"4e7b0082-74b4-43f3-bf68-d67397801800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"
Oct 30 19:54:14.213086 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.213061    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e7b0082-74b4-43f3-bf68-d67397801800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70893d69-zhqzc\" (UID: \"4e7b0082-74b4-43f3-bf68-d67397801800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"
Oct 30 19:54:14.214750 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.214726    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e7b0082-74b4-43f3-bf68-d67397801800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70893d69-zhqzc\" (UID: \"4e7b0082-74b4-43f3-bf68-d67397801800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"
Oct 30 19:54:14.224986 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.224963    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqwgd\" (UniqueName: \"kubernetes.io/projected/4e7b0082-74b4-43f3-bf68-d67397801800-kube-api-access-zqwgd\") pod \"maas-api-revoke-sub-70893d69-zhqzc\" (UID: \"4e7b0082-74b4-43f3-bf68-d67397801800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"
Oct 30 19:54:14.349581 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.349551    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"
Oct 30 19:54:14.651521 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:14.651488    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"]
Oct 30 19:54:14.651521 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:54:14.651513    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7b0082_74b4_43f3_bf68_d67397801800.slice/crio-5fc7740ff51cdaefe16b37331597f552d56dbf8edf80df3217a13b71a3e95811 WatchSource:0}: Error finding container 5fc7740ff51cdaefe16b37331597f552d56dbf8edf80df3217a13b71a3e95811: Status 404 returned error can't find the container with id 5fc7740ff51cdaefe16b37331597f552d56dbf8edf80df3217a13b71a3e95811
Oct 30 19:54:15.110404 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:15.110371    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc" event={"ID":"4e7b0082-74b4-43f3-bf68-d67397801800","Type":"ContainerStarted","Data":"ebf55bb429f9afadf14fa34934e19c1ab994e3f7439c7b3263ecfd93b2cedf2f"}
Oct 30 19:54:15.110929 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:15.110427    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc" event={"ID":"4e7b0082-74b4-43f3-bf68-d67397801800","Type":"ContainerStarted","Data":"5fc7740ff51cdaefe16b37331597f552d56dbf8edf80df3217a13b71a3e95811"}
Oct 30 19:54:15.112160 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:15.112127    4823 generic.go:356] "Generic (PLEG): container finished" podID="e34658db-8f32-48b4-a690-ff7fac03644c" containerID="d63b7fddd7578d23f6fafbd211d8d6e56b4fb0460dd1504ba0440ac237a9d21f" exitCode=0
Oct 30 19:54:15.112286 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:15.112208    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5" event={"ID":"e34658db-8f32-48b4-a690-ff7fac03644c","Type":"ContainerDied","Data":"d63b7fddd7578d23f6fafbd211d8d6e56b4fb0460dd1504ba0440ac237a9d21f"}
Oct 30 19:54:15.113907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:15.113886    4823 generic.go:356] "Generic (PLEG): container finished" podID="d37f565f-b457-42f3-9854-5f94bc8d60f2" containerID="bb815c521bf1aea059dc6cc4dbb99aeac9ab3000b12861f5b14035818e082aec" exitCode=0
Oct 30 19:54:15.114013 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:15.113959    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k" event={"ID":"d37f565f-b457-42f3-9854-5f94bc8d60f2","Type":"ContainerDied","Data":"bb815c521bf1aea059dc6cc4dbb99aeac9ab3000b12861f5b14035818e082aec"}
Oct 30 19:54:15.128343 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:15.128245    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc" podStartSLOduration=1.128232205 podStartE2EDuration="1.128232205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:15.127461853 +0000 UTC m=+1455.020076177" watchObservedRunningTime="2026-10-30 19:54:15.128232205 +0000 UTC m=+1455.020846557"
Oct 30 19:54:16.125347 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.125317    4823 generic.go:356] "Generic (PLEG): container finished" podID="4e7b0082-74b4-43f3-bf68-d67397801800" containerID="ebf55bb429f9afadf14fa34934e19c1ab994e3f7439c7b3263ecfd93b2cedf2f" exitCode=0
Oct 30 19:54:16.125794 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.125377    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc" event={"ID":"4e7b0082-74b4-43f3-bf68-d67397801800","Type":"ContainerDied","Data":"ebf55bb429f9afadf14fa34934e19c1ab994e3f7439c7b3263ecfd93b2cedf2f"}
Oct 30 19:54:16.331907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.331843    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"
Oct 30 19:54:16.345822 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.345797    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"
Oct 30 19:54:16.482936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.478274    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37f565f-b457-42f3-9854-5f94bc8d60f2-kube-api-access-zrwt7\" (UniqueName: \"kubernetes.io/projected/d37f565f-b457-42f3-9854-5f94bc8d60f2-kube-api-access-zrwt7\") pod \"d37f565f-b457-42f3-9854-5f94bc8d60f2\" (UID: \"d37f565f-b457-42f3-9854-5f94bc8d60f2\") "
Oct 30 19:54:16.482936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.478326    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d37f565f-b457-42f3-9854-5f94bc8d60f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37f565f-b457-42f3-9854-5f94bc8d60f2-maas-api-openshift-ca\") pod \"d37f565f-b457-42f3-9854-5f94bc8d60f2\" (UID: \"d37f565f-b457-42f3-9854-5f94bc8d60f2\") "
Oct 30 19:54:16.482936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.478411    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e34658db-8f32-48b4-a690-ff7fac03644c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34658db-8f32-48b4-a690-ff7fac03644c-maas-api-openshift-ca\") pod \"e34658db-8f32-48b4-a690-ff7fac03644c\" (UID: \"e34658db-8f32-48b4-a690-ff7fac03644c\") "
Oct 30 19:54:16.482936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.478487    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e34658db-8f32-48b4-a690-ff7fac03644c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34658db-8f32-48b4-a690-ff7fac03644c-maas-api-xks-ca\") pod \"e34658db-8f32-48b4-a690-ff7fac03644c\" (UID: \"e34658db-8f32-48b4-a690-ff7fac03644c\") "
Oct 30 19:54:16.482936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.478881    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e34658db-8f32-48b4-a690-ff7fac03644c-kube-api-access-x9z4w\" (UniqueName: \"kubernetes.io/projected/e34658db-8f32-48b4-a690-ff7fac03644c-kube-api-access-x9z4w\") pod \"e34658db-8f32-48b4-a690-ff7fac03644c\" (UID: \"e34658db-8f32-48b4-a690-ff7fac03644c\") "
Oct 30 19:54:16.482936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.479096    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d37f565f-b457-42f3-9854-5f94bc8d60f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37f565f-b457-42f3-9854-5f94bc8d60f2-maas-api-xks-ca\") pod \"d37f565f-b457-42f3-9854-5f94bc8d60f2\" (UID: \"d37f565f-b457-42f3-9854-5f94bc8d60f2\") "
Oct 30 19:54:16.482936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.480341    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34658db-8f32-48b4-a690-ff7fac03644c-maas-api-openshift-ca" pod "e34658db-8f32-48b4-a690-ff7fac03644c" (UID: "e34658db-8f32-48b4-a690-ff7fac03644c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:16.482936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.480623    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37f565f-b457-42f3-9854-5f94bc8d60f2-maas-api-openshift-ca" pod "d37f565f-b457-42f3-9854-5f94bc8d60f2" (UID: "d37f565f-b457-42f3-9854-5f94bc8d60f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:16.499972 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.499597    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37f565f-b457-42f3-9854-5f94bc8d60f2-kube-api-access-zrwt7" pod "d37f565f-b457-42f3-9854-5f94bc8d60f2" (UID: "d37f565f-b457-42f3-9854-5f94bc8d60f2"). InnerVolumeSpecName "kube-api-access-zrwt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:16.499972 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.499755    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d37f565f-b457-42f3-9854-5f94bc8d60f2-maas-api-xks-ca" pod "d37f565f-b457-42f3-9854-5f94bc8d60f2" (UID: "d37f565f-b457-42f3-9854-5f94bc8d60f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:16.502050 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.501873    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34658db-8f32-48b4-a690-ff7fac03644c-kube-api-access-x9z4w" pod "e34658db-8f32-48b4-a690-ff7fac03644c" (UID: "e34658db-8f32-48b4-a690-ff7fac03644c"). InnerVolumeSpecName "kube-api-access-x9z4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:16.502209 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.502046    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e34658db-8f32-48b4-a690-ff7fac03644c-maas-api-xks-ca" pod "e34658db-8f32-48b4-a690-ff7fac03644c" (UID: "e34658db-8f32-48b4-a690-ff7fac03644c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:16.580749 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.580719    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37f565f-b457-42f3-9854-5f94bc8d60f2-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:16.580749 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.580748    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrwt7\" (UniqueName: \"kubernetes.io/projected/d37f565f-b457-42f3-9854-5f94bc8d60f2-kube-api-access-zrwt7\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:16.580980 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.580758    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37f565f-b457-42f3-9854-5f94bc8d60f2-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:16.580980 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.580768    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34658db-8f32-48b4-a690-ff7fac03644c-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:16.580980 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.580777    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34658db-8f32-48b4-a690-ff7fac03644c-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:16.580980 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:16.580786    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9z4w\" (UniqueName: \"kubernetes.io/projected/e34658db-8f32-48b4-a690-ff7fac03644c-kube-api-access-x9z4w\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:17.134278 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:17.134242    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"
Oct 30 19:54:17.134705 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:17.134287    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5" event={"ID":"e34658db-8f32-48b4-a690-ff7fac03644c","Type":"ContainerDied","Data":"0e07a0722d8e22760e5245425f1f9af379e91f14ee212295a52aa8b54efdad43"}
Oct 30 19:54:17.134705 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:17.134332    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e07a0722d8e22760e5245425f1f9af379e91f14ee212295a52aa8b54efdad43"
Oct 30 19:54:17.135819 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:17.135797    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"
Oct 30 19:54:17.135954 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:17.135825    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k" event={"ID":"d37f565f-b457-42f3-9854-5f94bc8d60f2","Type":"ContainerDied","Data":"752cc9c01e28a394fee4968ee834a62b260288efc45d41e4c3f48f3195cde580"}
Oct 30 19:54:17.135954 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:17.135887    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="752cc9c01e28a394fee4968ee834a62b260288efc45d41e4c3f48f3195cde580"
Oct 30 19:54:17.298106 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:17.298076    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"
Oct 30 19:54:17.402980 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:17.398320    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e7b0082-74b4-43f3-bf68-d67397801800-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e7b0082-74b4-43f3-bf68-d67397801800-maas-api-xks-ca\") pod \"4e7b0082-74b4-43f3-bf68-d67397801800\" (UID: \"4e7b0082-74b4-43f3-bf68-d67397801800\") "
Oct 30 19:54:17.402980 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:17.398417    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7b0082-74b4-43f3-bf68-d67397801800-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e7b0082-74b4-43f3-bf68-d67397801800-maas-api-openshift-ca\") pod \"4e7b0082-74b4-43f3-bf68-d67397801800\" (UID: \"4e7b0082-74b4-43f3-bf68-d67397801800\") "
Oct 30 19:54:17.402980 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:17.398571    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7b0082-74b4-43f3-bf68-d67397801800-kube-api-access-zqwgd\" (UniqueName: \"kubernetes.io/projected/4e7b0082-74b4-43f3-bf68-d67397801800-kube-api-access-zqwgd\") pod \"4e7b0082-74b4-43f3-bf68-d67397801800\" (UID: \"4e7b0082-74b4-43f3-bf68-d67397801800\") "
Oct 30 19:54:17.402980 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:17.399845    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7b0082-74b4-43f3-bf68-d67397801800-maas-api-openshift-ca" pod "4e7b0082-74b4-43f3-bf68-d67397801800" (UID: "4e7b0082-74b4-43f3-bf68-d67397801800"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:17.402980 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:17.402664    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e7b0082-74b4-43f3-bf68-d67397801800-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:17.405619 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:17.405246    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7b0082-74b4-43f3-bf68-d67397801800-maas-api-xks-ca" pod "4e7b0082-74b4-43f3-bf68-d67397801800" (UID: "4e7b0082-74b4-43f3-bf68-d67397801800"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:17.416046 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:17.410494    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7b0082-74b4-43f3-bf68-d67397801800-kube-api-access-zqwgd" pod "4e7b0082-74b4-43f3-bf68-d67397801800" (UID: "4e7b0082-74b4-43f3-bf68-d67397801800"). InnerVolumeSpecName "kube-api-access-zqwgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:17.504026 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:17.503986    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqwgd\" (UniqueName: \"kubernetes.io/projected/4e7b0082-74b4-43f3-bf68-d67397801800-kube-api-access-zqwgd\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:17.504026 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:17.504020    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e7b0082-74b4-43f3-bf68-d67397801800-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:18.145737 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:18.145708    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"
Oct 30 19:54:18.146115 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:18.145737    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc" event={"ID":"4e7b0082-74b4-43f3-bf68-d67397801800","Type":"ContainerDied","Data":"5fc7740ff51cdaefe16b37331597f552d56dbf8edf80df3217a13b71a3e95811"}
Oct 30 19:54:18.146115 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:18.145788    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fc7740ff51cdaefe16b37331597f552d56dbf8edf80df3217a13b71a3e95811"
Oct 30 19:54:19.183067 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.183029    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"]
Oct 30 19:54:19.183929 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.183906    4823 cpu_manager.go:414] "containerMap: removing container" podUID="d37f565f-b457-42f3-9854-5f94bc8d60f2" containerName="revoke-keys"
Oct 30 19:54:19.183929 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.183930    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37f565f-b457-42f3-9854-5f94bc8d60f2" containerName="revoke-keys"
Oct 30 19:54:19.184124 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.183973    4823 cpu_manager.go:414] "containerMap: removing container" podUID="4e7b0082-74b4-43f3-bf68-d67397801800" containerName="revoke-keys"
Oct 30 19:54:19.184124 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.183981    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7b0082-74b4-43f3-bf68-d67397801800" containerName="revoke-keys"
Oct 30 19:54:19.184124 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.184034    4823 cpu_manager.go:414] "containerMap: removing container" podUID="e34658db-8f32-48b4-a690-ff7fac03644c" containerName="revoke-keys"
Oct 30 19:54:19.184124 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.184042    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e34658db-8f32-48b4-a690-ff7fac03644c" containerName="revoke-keys"
Oct 30 19:54:19.184324 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.184212    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="e34658db-8f32-48b4-a690-ff7fac03644c" containerName="revoke-keys"
Oct 30 19:54:19.184324 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.184230    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37f565f-b457-42f3-9854-5f94bc8d60f2" containerName="revoke-keys"
Oct 30 19:54:19.184324 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.184242    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7b0082-74b4-43f3-bf68-d67397801800" containerName="revoke-keys"
Oct 30 19:54:19.188112 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.188093    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"
Oct 30 19:54:19.200809 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.199623    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:54:19.200809 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.199790    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:54:19.200809 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.199880    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:54:19.200809 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.199976    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:54:19.211100 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.209575    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"]
Oct 30 19:54:19.330549 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.330512    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767ad954-8f7c-481f-9d8a-2f832fe6e07a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d8805a6-vcr9n\" (UID: \"767ad954-8f7c-481f-9d8a-2f832fe6e07a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"
Oct 30 19:54:19.330728 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.330708    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lljj\" (UniqueName: \"kubernetes.io/projected/767ad954-8f7c-481f-9d8a-2f832fe6e07a-kube-api-access-9lljj\") pod \"maas-api-revoke-sub-2d8805a6-vcr9n\" (UID: \"767ad954-8f7c-481f-9d8a-2f832fe6e07a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"
Oct 30 19:54:19.330845 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.330831    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767ad954-8f7c-481f-9d8a-2f832fe6e07a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d8805a6-vcr9n\" (UID: \"767ad954-8f7c-481f-9d8a-2f832fe6e07a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"
Oct 30 19:54:19.432731 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.432678    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767ad954-8f7c-481f-9d8a-2f832fe6e07a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d8805a6-vcr9n\" (UID: \"767ad954-8f7c-481f-9d8a-2f832fe6e07a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"
Oct 30 19:54:19.433107 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.433045    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lljj\" (UniqueName: \"kubernetes.io/projected/767ad954-8f7c-481f-9d8a-2f832fe6e07a-kube-api-access-9lljj\") pod \"maas-api-revoke-sub-2d8805a6-vcr9n\" (UID: \"767ad954-8f7c-481f-9d8a-2f832fe6e07a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"
Oct 30 19:54:19.433462 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.433430    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767ad954-8f7c-481f-9d8a-2f832fe6e07a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d8805a6-vcr9n\" (UID: \"767ad954-8f7c-481f-9d8a-2f832fe6e07a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"
Oct 30 19:54:19.433604 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.433580    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767ad954-8f7c-481f-9d8a-2f832fe6e07a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d8805a6-vcr9n\" (UID: \"767ad954-8f7c-481f-9d8a-2f832fe6e07a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"
Oct 30 19:54:19.435700 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.435679    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767ad954-8f7c-481f-9d8a-2f832fe6e07a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d8805a6-vcr9n\" (UID: \"767ad954-8f7c-481f-9d8a-2f832fe6e07a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"
Oct 30 19:54:19.447892 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.447835    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lljj\" (UniqueName: \"kubernetes.io/projected/767ad954-8f7c-481f-9d8a-2f832fe6e07a-kube-api-access-9lljj\") pod \"maas-api-revoke-sub-2d8805a6-vcr9n\" (UID: \"767ad954-8f7c-481f-9d8a-2f832fe6e07a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"
Oct 30 19:54:19.514497 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.514462    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"
Oct 30 19:54:19.812023 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:19.811949    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"]
Oct 30 19:54:19.815836 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:54:19.815805    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767ad954_8f7c_481f_9d8a_2f832fe6e07a.slice/crio-88d5b54ded42619d5b4432cc13d16349766a658b829b31f428099471773ececf WatchSource:0}: Error finding container 88d5b54ded42619d5b4432cc13d16349766a658b829b31f428099471773ececf: Status 404 returned error can't find the container with id 88d5b54ded42619d5b4432cc13d16349766a658b829b31f428099471773ececf
Oct 30 19:54:20.163382 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:20.163355    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n" event={"ID":"767ad954-8f7c-481f-9d8a-2f832fe6e07a","Type":"ContainerStarted","Data":"88d5b54ded42619d5b4432cc13d16349766a658b829b31f428099471773ececf"}
Oct 30 19:54:21.173498 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:21.173460    4823 generic.go:356] "Generic (PLEG): container finished" podID="767ad954-8f7c-481f-9d8a-2f832fe6e07a" containerID="c309d0360d3813eef7253df26c65b6ef3f2347a5140a6efd92b210eda6871fc9" exitCode=0
Oct 30 19:54:21.173882 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:21.173536    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n" event={"ID":"767ad954-8f7c-481f-9d8a-2f832fe6e07a","Type":"ContainerDied","Data":"c309d0360d3813eef7253df26c65b6ef3f2347a5140a6efd92b210eda6871fc9"}
Oct 30 19:54:22.360221 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:22.360193    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"
Oct 30 19:54:22.389420 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:22.389380    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767ad954-8f7c-481f-9d8a-2f832fe6e07a-kube-api-access-9lljj\" (UniqueName: \"kubernetes.io/projected/767ad954-8f7c-481f-9d8a-2f832fe6e07a-kube-api-access-9lljj\") pod \"767ad954-8f7c-481f-9d8a-2f832fe6e07a\" (UID: \"767ad954-8f7c-481f-9d8a-2f832fe6e07a\") "
Oct 30 19:54:22.390473 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:22.390434    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/767ad954-8f7c-481f-9d8a-2f832fe6e07a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767ad954-8f7c-481f-9d8a-2f832fe6e07a-maas-api-xks-ca\") pod \"767ad954-8f7c-481f-9d8a-2f832fe6e07a\" (UID: \"767ad954-8f7c-481f-9d8a-2f832fe6e07a\") "
Oct 30 19:54:22.390763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:22.390746    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/767ad954-8f7c-481f-9d8a-2f832fe6e07a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767ad954-8f7c-481f-9d8a-2f832fe6e07a-maas-api-openshift-ca\") pod \"767ad954-8f7c-481f-9d8a-2f832fe6e07a\" (UID: \"767ad954-8f7c-481f-9d8a-2f832fe6e07a\") "
Oct 30 19:54:22.391212 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:22.391179    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767ad954-8f7c-481f-9d8a-2f832fe6e07a-maas-api-openshift-ca" pod "767ad954-8f7c-481f-9d8a-2f832fe6e07a" (UID: "767ad954-8f7c-481f-9d8a-2f832fe6e07a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:22.391317 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:22.391306    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/767ad954-8f7c-481f-9d8a-2f832fe6e07a-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:22.392208 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:22.392180    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767ad954-8f7c-481f-9d8a-2f832fe6e07a-kube-api-access-9lljj" pod "767ad954-8f7c-481f-9d8a-2f832fe6e07a" (UID: "767ad954-8f7c-481f-9d8a-2f832fe6e07a"). InnerVolumeSpecName "kube-api-access-9lljj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:22.392818 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:22.392787    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/767ad954-8f7c-481f-9d8a-2f832fe6e07a-maas-api-xks-ca" pod "767ad954-8f7c-481f-9d8a-2f832fe6e07a" (UID: "767ad954-8f7c-481f-9d8a-2f832fe6e07a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:22.492386 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:22.492296    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lljj\" (UniqueName: \"kubernetes.io/projected/767ad954-8f7c-481f-9d8a-2f832fe6e07a-kube-api-access-9lljj\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:22.492386 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:22.492335    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/767ad954-8f7c-481f-9d8a-2f832fe6e07a-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:23.190392 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:23.190361    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"
Oct 30 19:54:23.190598 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:23.190392    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n" event={"ID":"767ad954-8f7c-481f-9d8a-2f832fe6e07a","Type":"ContainerDied","Data":"88d5b54ded42619d5b4432cc13d16349766a658b829b31f428099471773ececf"}
Oct 30 19:54:23.190598 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:23.190441    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88d5b54ded42619d5b4432cc13d16349766a658b829b31f428099471773ececf"
Oct 30 19:54:24.281136 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.281108    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"]
Oct 30 19:54:24.281768 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.281754    4823 cpu_manager.go:414] "containerMap: removing container" podUID="767ad954-8f7c-481f-9d8a-2f832fe6e07a" containerName="revoke-keys"
Oct 30 19:54:24.281834 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.281769    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767ad954-8f7c-481f-9d8a-2f832fe6e07a" containerName="revoke-keys"
Oct 30 19:54:24.281905 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.281895    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="767ad954-8f7c-481f-9d8a-2f832fe6e07a" containerName="revoke-keys"
Oct 30 19:54:24.285941 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.285924    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"
Oct 30 19:54:24.295215 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.295194    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:54:24.295458 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.295381    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:54:24.295892 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.295615    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:54:24.295892 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.295695    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:54:24.298986 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.298963    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"]
Oct 30 19:54:24.317270 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.317237    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sjf6\" (UniqueName: \"kubernetes.io/projected/51289fa4-20a0-4ee8-85ba-f75ff11cb513-kube-api-access-8sjf6\") pod \"maas-api-revoke-sub-ebad7cfb-whkvk\" (UID: \"51289fa4-20a0-4ee8-85ba-f75ff11cb513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"
Oct 30 19:54:24.317407 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.317380    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51289fa4-20a0-4ee8-85ba-f75ff11cb513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebad7cfb-whkvk\" (UID: \"51289fa4-20a0-4ee8-85ba-f75ff11cb513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"
Oct 30 19:54:24.317549 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.317527    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51289fa4-20a0-4ee8-85ba-f75ff11cb513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebad7cfb-whkvk\" (UID: \"51289fa4-20a0-4ee8-85ba-f75ff11cb513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"
Oct 30 19:54:24.420382 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.420332    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sjf6\" (UniqueName: \"kubernetes.io/projected/51289fa4-20a0-4ee8-85ba-f75ff11cb513-kube-api-access-8sjf6\") pod \"maas-api-revoke-sub-ebad7cfb-whkvk\" (UID: \"51289fa4-20a0-4ee8-85ba-f75ff11cb513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"
Oct 30 19:54:24.420586 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.420486    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51289fa4-20a0-4ee8-85ba-f75ff11cb513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebad7cfb-whkvk\" (UID: \"51289fa4-20a0-4ee8-85ba-f75ff11cb513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"
Oct 30 19:54:24.420778 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.420670    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51289fa4-20a0-4ee8-85ba-f75ff11cb513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebad7cfb-whkvk\" (UID: \"51289fa4-20a0-4ee8-85ba-f75ff11cb513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"
Oct 30 19:54:24.448782 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.448069    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51289fa4-20a0-4ee8-85ba-f75ff11cb513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebad7cfb-whkvk\" (UID: \"51289fa4-20a0-4ee8-85ba-f75ff11cb513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"
Oct 30 19:54:24.448782 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.448532    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51289fa4-20a0-4ee8-85ba-f75ff11cb513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebad7cfb-whkvk\" (UID: \"51289fa4-20a0-4ee8-85ba-f75ff11cb513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"
Oct 30 19:54:24.463777 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.463742    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sjf6\" (UniqueName: \"kubernetes.io/projected/51289fa4-20a0-4ee8-85ba-f75ff11cb513-kube-api-access-8sjf6\") pod \"maas-api-revoke-sub-ebad7cfb-whkvk\" (UID: \"51289fa4-20a0-4ee8-85ba-f75ff11cb513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"
Oct 30 19:54:24.602714 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.602625    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"
Oct 30 19:54:24.897290 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:54:24.897260    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51289fa4_20a0_4ee8_85ba_f75ff11cb513.slice/crio-e42cf5315c5e464c2dfcbe1c6856f684fb4662b965d4f7605c1bf5c9a7215a40 WatchSource:0}: Error finding container e42cf5315c5e464c2dfcbe1c6856f684fb4662b965d4f7605c1bf5c9a7215a40: Status 404 returned error can't find the container with id e42cf5315c5e464c2dfcbe1c6856f684fb4662b965d4f7605c1bf5c9a7215a40
Oct 30 19:54:24.897290 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:24.897283    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"]
Oct 30 19:54:25.207641 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.207601    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk" event={"ID":"51289fa4-20a0-4ee8-85ba-f75ff11cb513","Type":"ContainerStarted","Data":"5283506b7e33e28a93eab2bb1d377bac34df5c486da5b428e1317d6fe3de4ac4"}
Oct 30 19:54:25.207803 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.207658    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk" event={"ID":"51289fa4-20a0-4ee8-85ba-f75ff11cb513","Type":"ContainerStarted","Data":"e42cf5315c5e464c2dfcbe1c6856f684fb4662b965d4f7605c1bf5c9a7215a40"}
Oct 30 19:54:25.240367 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.240304    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk" podStartSLOduration=1.240285909 podStartE2EDuration="1.240285909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:25.23366328 +0000 UTC m=+1465.126277605" watchObservedRunningTime="2026-10-30 19:54:25.240285909 +0000 UTC m=+1465.132900236"
Oct 30 19:54:25.357465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.357434    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"]
Oct 30 19:54:25.362244 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.362222    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"
Oct 30 19:54:25.394625 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.394596    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"]
Oct 30 19:54:25.434886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.434779    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-167a933b-894v5\" (UID: \"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"
Oct 30 19:54:25.435051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.434977    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5kp5\" (UniqueName: \"kubernetes.io/projected/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-kube-api-access-h5kp5\") pod \"maas-api-revoke-sub-167a933b-894v5\" (UID: \"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"
Oct 30 19:54:25.435117 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.435053    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-167a933b-894v5\" (UID: \"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"
Oct 30 19:54:25.536951 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.536914    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-167a933b-894v5\" (UID: \"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"
Oct 30 19:54:25.537098 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.537050    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5kp5\" (UniqueName: \"kubernetes.io/projected/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-kube-api-access-h5kp5\") pod \"maas-api-revoke-sub-167a933b-894v5\" (UID: \"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"
Oct 30 19:54:25.537167 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.537117    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-167a933b-894v5\" (UID: \"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"
Oct 30 19:54:25.537653 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.537631    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-167a933b-894v5\" (UID: \"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"
Oct 30 19:54:25.539659 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.539635    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-167a933b-894v5\" (UID: \"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"
Oct 30 19:54:25.563287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.563264    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5kp5\" (UniqueName: \"kubernetes.io/projected/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-kube-api-access-h5kp5\") pod \"maas-api-revoke-sub-167a933b-894v5\" (UID: \"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"
Oct 30 19:54:25.677272 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.677240    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"
Oct 30 19:54:25.970348 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:54:25.970307    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9fe8c4_6cfa_4b87_b0f8_a82f0cf4b736.slice/crio-ec43c5019ba43c16597b203ef7c949d608fde25eabef78e77c551274831b4667 WatchSource:0}: Error finding container ec43c5019ba43c16597b203ef7c949d608fde25eabef78e77c551274831b4667: Status 404 returned error can't find the container with id ec43c5019ba43c16597b203ef7c949d608fde25eabef78e77c551274831b4667
Oct 30 19:54:25.970830 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:25.970807    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"]
Oct 30 19:54:26.218222 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.218130    4823 generic.go:356] "Generic (PLEG): container finished" podID="51289fa4-20a0-4ee8-85ba-f75ff11cb513" containerID="5283506b7e33e28a93eab2bb1d377bac34df5c486da5b428e1317d6fe3de4ac4" exitCode=0
Oct 30 19:54:26.218222 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.218208    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk" event={"ID":"51289fa4-20a0-4ee8-85ba-f75ff11cb513","Type":"ContainerDied","Data":"5283506b7e33e28a93eab2bb1d377bac34df5c486da5b428e1317d6fe3de4ac4"}
Oct 30 19:54:26.219662 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.219638    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5" event={"ID":"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736","Type":"ContainerStarted","Data":"ec43c5019ba43c16597b203ef7c949d608fde25eabef78e77c551274831b4667"}
Oct 30 19:54:26.614686 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.614651    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"]
Oct 30 19:54:26.620138 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.620118    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.638593 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.638567    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"]
Oct 30 19:54:26.665652 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.665618    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.665830 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.665661    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.665963 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.665831    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.665963 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.665922    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.666079 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.666035    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.666203 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.666184    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csj8n\" (UniqueName: \"kubernetes.io/projected/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-kube-api-access-csj8n\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.666272 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.666223    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.666272 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.666248    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.666373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.666282    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.767527 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.767496    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.767716 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.767557    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.767716 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.767629    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.767716 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.767662    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csj8n\" (UniqueName: \"kubernetes.io/projected/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-kube-api-access-csj8n\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.767716 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.767688    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.767716 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.767706    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.768012 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.767739    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.768012 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.767803    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.768012 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.767828    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.768223 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.768205    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.768339 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.768313    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.768399 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.768335    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.768399 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.768381    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.768487 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.768461    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.770099 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.770079    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.770230 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.770174    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.787420 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.787393    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.787738 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.787720    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csj8n\" (UniqueName: \"kubernetes.io/projected/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-kube-api-access-csj8n\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:26.936426 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:26.936385    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:27.232201 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:27.232104    4823 generic.go:356] "Generic (PLEG): container finished" podID="6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736" containerID="95201873bc7906360856f1555267fa164c89e195a5abd2044036bada28bdc1ab" exitCode=0
Oct 30 19:54:27.232360 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:27.232193    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5" event={"ID":"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736","Type":"ContainerDied","Data":"95201873bc7906360856f1555267fa164c89e195a5abd2044036bada28bdc1ab"}
Oct 30 19:54:27.247490 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:54:27.247463    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf2ad6b_cfdb_4e4a_99e5_1021675227c2.slice/crio-61d16eea8dfe16968d4a6fe41ea883810e9cf4c6954e0746cdfaaf3d18fcdcad WatchSource:0}: Error finding container 61d16eea8dfe16968d4a6fe41ea883810e9cf4c6954e0746cdfaaf3d18fcdcad: Status 404 returned error can't find the container with id 61d16eea8dfe16968d4a6fe41ea883810e9cf4c6954e0746cdfaaf3d18fcdcad
Oct 30 19:54:27.247915 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:27.247887    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"]
Oct 30 19:54:27.250271 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:27.250238    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:27.250351 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:27.250301    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:27.250351 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:27.250332    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:27.401910 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:27.401881    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"
Oct 30 19:54:27.479344 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:27.479303    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51289fa4-20a0-4ee8-85ba-f75ff11cb513-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51289fa4-20a0-4ee8-85ba-f75ff11cb513-maas-api-openshift-ca\") pod \"51289fa4-20a0-4ee8-85ba-f75ff11cb513\" (UID: \"51289fa4-20a0-4ee8-85ba-f75ff11cb513\") "
Oct 30 19:54:27.479671 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:27.479651    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51289fa4-20a0-4ee8-85ba-f75ff11cb513-kube-api-access-8sjf6\" (UniqueName: \"kubernetes.io/projected/51289fa4-20a0-4ee8-85ba-f75ff11cb513-kube-api-access-8sjf6\") pod \"51289fa4-20a0-4ee8-85ba-f75ff11cb513\" (UID: \"51289fa4-20a0-4ee8-85ba-f75ff11cb513\") "
Oct 30 19:54:27.479757 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:27.479728    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51289fa4-20a0-4ee8-85ba-f75ff11cb513-maas-api-openshift-ca" pod "51289fa4-20a0-4ee8-85ba-f75ff11cb513" (UID: "51289fa4-20a0-4ee8-85ba-f75ff11cb513"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:27.479794 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:27.479740    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51289fa4-20a0-4ee8-85ba-f75ff11cb513-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51289fa4-20a0-4ee8-85ba-f75ff11cb513-maas-api-xks-ca\") pod \"51289fa4-20a0-4ee8-85ba-f75ff11cb513\" (UID: \"51289fa4-20a0-4ee8-85ba-f75ff11cb513\") "
Oct 30 19:54:27.480489 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:27.480459    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51289fa4-20a0-4ee8-85ba-f75ff11cb513-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.482087 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:27.482055    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51289fa4-20a0-4ee8-85ba-f75ff11cb513-maas-api-xks-ca" pod "51289fa4-20a0-4ee8-85ba-f75ff11cb513" (UID: "51289fa4-20a0-4ee8-85ba-f75ff11cb513"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:27.482934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:27.482836    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51289fa4-20a0-4ee8-85ba-f75ff11cb513-kube-api-access-8sjf6" pod "51289fa4-20a0-4ee8-85ba-f75ff11cb513" (UID: "51289fa4-20a0-4ee8-85ba-f75ff11cb513"). InnerVolumeSpecName "kube-api-access-8sjf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:27.596073 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:27.596036    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sjf6\" (UniqueName: \"kubernetes.io/projected/51289fa4-20a0-4ee8-85ba-f75ff11cb513-kube-api-access-8sjf6\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.596073 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:27.596076    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51289fa4-20a0-4ee8-85ba-f75ff11cb513-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:28.243037 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.242998    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz" event={"ID":"faf2ad6b-cfdb-4e4a-99e5-1021675227c2","Type":"ContainerStarted","Data":"670106df987067313419fb56598633b9fd80dc22f729fcd5915de7a217bf1c97"}
Oct 30 19:54:28.243482 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.243063    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz" event={"ID":"faf2ad6b-cfdb-4e4a-99e5-1021675227c2","Type":"ContainerStarted","Data":"61d16eea8dfe16968d4a6fe41ea883810e9cf4c6954e0746cdfaaf3d18fcdcad"}
Oct 30 19:54:28.244693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.244671    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"
Oct 30 19:54:28.244824 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.244705    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk" event={"ID":"51289fa4-20a0-4ee8-85ba-f75ff11cb513","Type":"ContainerDied","Data":"e42cf5315c5e464c2dfcbe1c6856f684fb4662b965d4f7605c1bf5c9a7215a40"}
Oct 30 19:54:28.244824 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.244742    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e42cf5315c5e464c2dfcbe1c6856f684fb4662b965d4f7605c1bf5c9a7215a40"
Oct 30 19:54:28.290838 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.290777    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz" podStartSLOduration=2.290756051 podStartE2EDuration="2.290756051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:28.287787647 +0000 UTC m=+1468.180402000" watchObservedRunningTime="2026-10-30 19:54:28.290756051 +0000 UTC m=+1468.183370377"
Oct 30 19:54:28.494508 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.494444    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"
Oct 30 19:54:28.613259 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.613224    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-maas-api-xks-ca\") pod \"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736\" (UID: \"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736\") "
Oct 30 19:54:28.613424 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.613334    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-kube-api-access-h5kp5\" (UniqueName: \"kubernetes.io/projected/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-kube-api-access-h5kp5\") pod \"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736\" (UID: \"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736\") "
Oct 30 19:54:28.613497 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.613436    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-maas-api-openshift-ca\") pod \"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736\" (UID: \"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736\") "
Oct 30 19:54:28.613847 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.613816    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-maas-api-openshift-ca" pod "6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736" (UID: "6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:28.614392 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.614361    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:28.615654 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.615626    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-maas-api-xks-ca" pod "6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736" (UID: "6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:28.616451 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.616425    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-kube-api-access-h5kp5" pod "6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736" (UID: "6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736"). InnerVolumeSpecName "kube-api-access-h5kp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:28.716048 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.716013    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:28.716048 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.716045    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5kp5\" (UniqueName: \"kubernetes.io/projected/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736-kube-api-access-h5kp5\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:28.937313 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.937272    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:28.940208 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.940175    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:54:28.940481 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:28.940447    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz" podUID="faf2ad6b-cfdb-4e4a-99e5-1021675227c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:54:29.268937 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.265787    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"
Oct 30 19:54:29.276950 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.273022    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5" event={"ID":"6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736","Type":"ContainerDied","Data":"ec43c5019ba43c16597b203ef7c949d608fde25eabef78e77c551274831b4667"}
Oct 30 19:54:29.276950 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.273087    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec43c5019ba43c16597b203ef7c949d608fde25eabef78e77c551274831b4667"
Oct 30 19:54:29.734315 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.734278    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"]
Oct 30 19:54:29.735000 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.734984    4823 cpu_manager.go:414] "containerMap: removing container" podUID="51289fa4-20a0-4ee8-85ba-f75ff11cb513" containerName="revoke-keys"
Oct 30 19:54:29.735043 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.735001    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51289fa4-20a0-4ee8-85ba-f75ff11cb513" containerName="revoke-keys"
Oct 30 19:54:29.735043 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.735023    4823 cpu_manager.go:414] "containerMap: removing container" podUID="6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736" containerName="revoke-keys"
Oct 30 19:54:29.735043 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.735028    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736" containerName="revoke-keys"
Oct 30 19:54:29.735159 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.735149    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="51289fa4-20a0-4ee8-85ba-f75ff11cb513" containerName="revoke-keys"
Oct 30 19:54:29.735194 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.735163    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736" containerName="revoke-keys"
Oct 30 19:54:29.739375 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.739359    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"
Oct 30 19:54:29.743728 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.743702    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:54:29.744112 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.744091    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:54:29.744603 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.744573    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:54:29.744792 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.744768    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:54:29.751446 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.751421    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"]
Oct 30 19:54:29.832603 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.832557    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0c7d84-3c20-48bb-908c-3624111e0abb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3d2ad17-xdc75\" (UID: \"6e0c7d84-3c20-48bb-908c-3624111e0abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"
Oct 30 19:54:29.832788 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.832685    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55r8z\" (UniqueName: \"kubernetes.io/projected/6e0c7d84-3c20-48bb-908c-3624111e0abb-kube-api-access-55r8z\") pod \"maas-api-revoke-sub-c3d2ad17-xdc75\" (UID: \"6e0c7d84-3c20-48bb-908c-3624111e0abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"
Oct 30 19:54:29.832788 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.832721    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0c7d84-3c20-48bb-908c-3624111e0abb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3d2ad17-xdc75\" (UID: \"6e0c7d84-3c20-48bb-908c-3624111e0abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"
Oct 30 19:54:29.934591 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.934554    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0c7d84-3c20-48bb-908c-3624111e0abb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3d2ad17-xdc75\" (UID: \"6e0c7d84-3c20-48bb-908c-3624111e0abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"
Oct 30 19:54:29.934787 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.934641    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55r8z\" (UniqueName: \"kubernetes.io/projected/6e0c7d84-3c20-48bb-908c-3624111e0abb-kube-api-access-55r8z\") pod \"maas-api-revoke-sub-c3d2ad17-xdc75\" (UID: \"6e0c7d84-3c20-48bb-908c-3624111e0abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"
Oct 30 19:54:29.934787 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.934671    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0c7d84-3c20-48bb-908c-3624111e0abb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3d2ad17-xdc75\" (UID: \"6e0c7d84-3c20-48bb-908c-3624111e0abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"
Oct 30 19:54:29.935483 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.935456    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0c7d84-3c20-48bb-908c-3624111e0abb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3d2ad17-xdc75\" (UID: \"6e0c7d84-3c20-48bb-908c-3624111e0abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"
Oct 30 19:54:29.937308 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.937279    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0c7d84-3c20-48bb-908c-3624111e0abb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3d2ad17-xdc75\" (UID: \"6e0c7d84-3c20-48bb-908c-3624111e0abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"
Oct 30 19:54:29.937308 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.937292    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:54:29.937478 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.937346    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz" podUID="faf2ad6b-cfdb-4e4a-99e5-1021675227c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:54:29.948151 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:29.948122    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55r8z\" (UniqueName: \"kubernetes.io/projected/6e0c7d84-3c20-48bb-908c-3624111e0abb-kube-api-access-55r8z\") pod \"maas-api-revoke-sub-c3d2ad17-xdc75\" (UID: \"6e0c7d84-3c20-48bb-908c-3624111e0abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"
Oct 30 19:54:30.076930 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:30.073194    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"
Oct 30 19:54:30.385219 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:30.385192    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"]
Oct 30 19:54:30.936675 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:30.936632    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:54:30.936850 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:30.936689    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz" podUID="faf2ad6b-cfdb-4e4a-99e5-1021675227c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:54:31.301745 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:31.301697    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75" event={"ID":"6e0c7d84-3c20-48bb-908c-3624111e0abb","Type":"ContainerStarted","Data":"138ac8357d065bbe6a1eb05df0c58b9906ec00a7547352d2f6ceac27c02090e9"}
Oct 30 19:54:31.301970 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:31.301764    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75" event={"ID":"6e0c7d84-3c20-48bb-908c-3624111e0abb","Type":"ContainerStarted","Data":"b02d386ea07a61be8c3325530b73112417f3c36b4201d690eb1cacf9eb90dccd"}
Oct 30 19:54:31.328449 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:31.328398    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75" podStartSLOduration=2.328382818 podStartE2EDuration="2.328382818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:31.327697703 +0000 UTC m=+1471.220312027" watchObservedRunningTime="2026-10-30 19:54:31.328382818 +0000 UTC m=+1471.220997154"
Oct 30 19:54:31.936901 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:31.936831    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:54:31.937363 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:31.936922    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz" podUID="faf2ad6b-cfdb-4e4a-99e5-1021675227c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:54:32.035990 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:54:32.035927    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:54:32.310999 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:32.310914    4823 generic.go:356] "Generic (PLEG): container finished" podID="6e0c7d84-3c20-48bb-908c-3624111e0abb" containerID="138ac8357d065bbe6a1eb05df0c58b9906ec00a7547352d2f6ceac27c02090e9" exitCode=0
Oct 30 19:54:32.311153 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:32.311004    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75" event={"ID":"6e0c7d84-3c20-48bb-908c-3624111e0abb","Type":"ContainerDied","Data":"138ac8357d065bbe6a1eb05df0c58b9906ec00a7547352d2f6ceac27c02090e9"}
Oct 30 19:54:32.937168 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:32.937134    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:54:32.937527 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:32.937186    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz" podUID="faf2ad6b-cfdb-4e4a-99e5-1021675227c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:54:33.600515 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:33.600485    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"
Oct 30 19:54:33.731489 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:33.731452    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e0c7d84-3c20-48bb-908c-3624111e0abb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0c7d84-3c20-48bb-908c-3624111e0abb-maas-api-openshift-ca\") pod \"6e0c7d84-3c20-48bb-908c-3624111e0abb\" (UID: \"6e0c7d84-3c20-48bb-908c-3624111e0abb\") "
Oct 30 19:54:33.731697 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:33.731678    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e0c7d84-3c20-48bb-908c-3624111e0abb-kube-api-access-55r8z\" (UniqueName: \"kubernetes.io/projected/6e0c7d84-3c20-48bb-908c-3624111e0abb-kube-api-access-55r8z\") pod \"6e0c7d84-3c20-48bb-908c-3624111e0abb\" (UID: \"6e0c7d84-3c20-48bb-908c-3624111e0abb\") "
Oct 30 19:54:33.731796 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:33.731772    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0c7d84-3c20-48bb-908c-3624111e0abb-maas-api-openshift-ca" pod "6e0c7d84-3c20-48bb-908c-3624111e0abb" (UID: "6e0c7d84-3c20-48bb-908c-3624111e0abb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:33.731873 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:33.731834    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e0c7d84-3c20-48bb-908c-3624111e0abb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0c7d84-3c20-48bb-908c-3624111e0abb-maas-api-xks-ca\") pod \"6e0c7d84-3c20-48bb-908c-3624111e0abb\" (UID: \"6e0c7d84-3c20-48bb-908c-3624111e0abb\") "
Oct 30 19:54:33.732399 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:33.732377    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0c7d84-3c20-48bb-908c-3624111e0abb-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:33.733698 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:33.733675    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e0c7d84-3c20-48bb-908c-3624111e0abb-maas-api-xks-ca" pod "6e0c7d84-3c20-48bb-908c-3624111e0abb" (UID: "6e0c7d84-3c20-48bb-908c-3624111e0abb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:33.733777 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:33.733752    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0c7d84-3c20-48bb-908c-3624111e0abb-kube-api-access-55r8z" pod "6e0c7d84-3c20-48bb-908c-3624111e0abb" (UID: "6e0c7d84-3c20-48bb-908c-3624111e0abb"). InnerVolumeSpecName "kube-api-access-55r8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:33.834445 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:33.834364    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55r8z\" (UniqueName: \"kubernetes.io/projected/6e0c7d84-3c20-48bb-908c-3624111e0abb-kube-api-access-55r8z\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:33.834445 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:33.834397    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0c7d84-3c20-48bb-908c-3624111e0abb-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:33.937243 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:33.937211    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:54:33.937611 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:33.937270    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz" podUID="faf2ad6b-cfdb-4e4a-99e5-1021675227c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:54:34.335038 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:34.335007    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"
Oct 30 19:54:34.335305 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:34.335016    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75" event={"ID":"6e0c7d84-3c20-48bb-908c-3624111e0abb","Type":"ContainerDied","Data":"b02d386ea07a61be8c3325530b73112417f3c36b4201d690eb1cacf9eb90dccd"}
Oct 30 19:54:34.335402 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:34.335342    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b02d386ea07a61be8c3325530b73112417f3c36b4201d690eb1cacf9eb90dccd"
Oct 30 19:54:34.941085 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:34.941053    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:34.945304 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:34.943205    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:34.952715 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:34.952685    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 19:54:36.629349 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:36.629316    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:54:39.167886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:39.167024    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:39.362886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:39.362052    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:39.453155 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:39.453049    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:39.570055 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:39.569985    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:39.618464 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:39.603350    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:39.722945 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:39.721999    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Oct 30 19:54:39.790892 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:39.783890    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:39.790892 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:39.785243    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:54:39.862887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:39.861903    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"]
Oct 30 19:54:39.863084 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:39.863058    4823 cpu_manager.go:414] "containerMap: removing container" podUID="6e0c7d84-3c20-48bb-908c-3624111e0abb" containerName="revoke-keys"
Oct 30 19:54:39.863084 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:39.863078    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e0c7d84-3c20-48bb-908c-3624111e0abb" containerName="revoke-keys"
Oct 30 19:54:39.887886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:39.863294    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e0c7d84-3c20-48bb-908c-3624111e0abb" containerName="revoke-keys"
Oct 30 19:54:40.046987 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.021093    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:54:40.073907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.050109    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:54:40.193935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.168619    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:40.193935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.183151    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:54:40.193935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.184309    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.240344 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.223397    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-df345a65-openshift-default-dockercfg-9b65m\""
Oct 30 19:54:40.325979 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.251991    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:40.325979 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.282518    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-istio-data\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.325979 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.282625    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-workload-socket\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.325979 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.282669    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgrj4\" (UniqueName: \"kubernetes.io/projected/0947f714-d384-4e21-829a-cb2ab5f70636-kube-api-access-zgrj4\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.325979 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.282719    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-workload-certs\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.325979 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.282786    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0947f714-d384-4e21-829a-cb2ab5f70636-istio-podinfo\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.325979 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.282844    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-istio-envoy\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.339211 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.339173    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0947f714-d384-4e21-829a-cb2ab5f70636-istiod-ca-cert\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.339534 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.339515    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-credential-socket\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.339752 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.339736    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0947f714-d384-4e21-829a-cb2ab5f70636-istio-token\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.365675 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.364954    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:40.619058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.475138    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:40.619058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.579150    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:40.619058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.598728    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0947f714-d384-4e21-829a-cb2ab5f70636-istio-podinfo\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.619058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.598799    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-istio-envoy\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.619058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.598883    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0947f714-d384-4e21-829a-cb2ab5f70636-istiod-ca-cert\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.619058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.598918    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-credential-socket\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.619058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.598998    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0947f714-d384-4e21-829a-cb2ab5f70636-istio-token\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.619058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.614709    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0947f714-d384-4e21-829a-cb2ab5f70636-istiod-ca-cert\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.619058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.614941    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-istio-data\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.619058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.615028    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-workload-socket\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.619058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.615089    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgrj4\" (UniqueName: \"kubernetes.io/projected/0947f714-d384-4e21-829a-cb2ab5f70636-kube-api-access-zgrj4\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.619058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.615156    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-workload-certs\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.619058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.615554    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-workload-certs\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.619058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.615794    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-credential-socket\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.733888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.703251    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-istio-data\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.733888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.704030    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-workload-socket\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.835890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.810966    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:40.862889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.837644    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:40.979949 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.950315    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-istio-envoy\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:40.979949 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:40.951382    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0947f714-d384-4e21-829a-cb2ab5f70636-istio-token\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:41.278959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.133082    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:54:41.278959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.136645    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:54:41.278959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.161192    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"]
Oct 30 19:54:41.278959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.208791    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:54:41.278959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.208885    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:54:41.278959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.208926    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:54:41.278959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.208960    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:54:41.278959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.208992    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:54:41.278959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.209024    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:54:41.278959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.209057    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:54:41.278959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.216390    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:41.288886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.280135    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:41.288886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.280494    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:54:41.463903 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.448008    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:41.488009 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.487957    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:41.577935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.574702    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgrj4\" (UniqueName: \"kubernetes.io/projected/0947f714-d384-4e21-829a-cb2ab5f70636-kube-api-access-zgrj4\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:41.632887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.599906    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0947f714-d384-4e21-829a-cb2ab5f70636-istio-podinfo\") pod \"e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") " pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:41.687894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:41.673145    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:42.116770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.094005    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"]
Oct 30 19:54:42.233653 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.233531    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:54:42.246988 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.234927    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:42.246988 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.234961    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:42.246988 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.237924    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:54:42.246988 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.235003    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:54:42.246988 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.235021    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:54:42.270064 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.234983    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:42.444888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.334374    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:54:42.444888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.387896    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.555225 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.507075    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:42.655887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.570767    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.668889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.664995    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.714149 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.714102    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.714506 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.714482    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.714744 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.714729    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181fb3fa-5478-4c39-9cfe-de04f2070e49-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.714881 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.714850    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9tq8\" (UniqueName: \"kubernetes.io/projected/181fb3fa-5478-4c39-9cfe-de04f2070e49-kube-api-access-t9tq8\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.715029 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.715012    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.740887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.715156    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.741529 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.741498    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.868334 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.868295    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.868706 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.868674    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.868961 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.868939    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.869125 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.869108    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.869316 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.869298    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181fb3fa-5478-4c39-9cfe-de04f2070e49-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.869456 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.869438    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9tq8\" (UniqueName: \"kubernetes.io/projected/181fb3fa-5478-4c39-9cfe-de04f2070e49-kube-api-access-t9tq8\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.869584 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.869567    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.869720 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.869702    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.870426 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.870403    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.870817 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.870797    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.873963 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.873935    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181fb3fa-5478-4c39-9cfe-de04f2070e49-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.874771 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.874749    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.875200 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.875163    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded"
Oct 30 19:54:42.875330 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.875309    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:42.907888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.895653    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"]
Oct 30 19:54:42.974890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:42.974759    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:43.005885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:43.002273    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:43.330291 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:43.276120    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:43.330291 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:43.301146    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:54:43.330291 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:43.301418    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:54:43.330291 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:43.306759    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:43.330291 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:43.313050    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:43.330291 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:43.313339    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:43.330291 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:43.314456    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:54:43.387011 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:43.386958    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"]
Oct 30 19:54:43.666920 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:43.655254    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9tq8\" (UniqueName: \"kubernetes.io/projected/181fb3fa-5478-4c39-9cfe-de04f2070e49-kube-api-access-t9tq8\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:44.195474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:43.772385    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded"
Oct 30 19:54:44.195474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:43.842082    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:44.195474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:43.862975    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:44.195474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:43.889824    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.195474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.089641    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:44.195474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.089848    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:54:44.195474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.167363    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"]
Oct 30 19:54:44.244520 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.236893    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:54:44.259100 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.257691    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:44.291976 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.291947    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:54:44.292156 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.292018    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:54:44.362923 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.361975    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:54:44.362923 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.362017    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"]
Oct 30 19:54:44.364675 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.364340    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.364675 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.364455    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.364675 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.364506    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrw4b\" (UniqueName: \"kubernetes.io/projected/9d0dea3a-53d6-425f-a202-506d89080874-kube-api-access-lrw4b\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.364675 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.364565    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0dea3a-53d6-425f-a202-506d89080874-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.365977 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.364961    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0dea3a-53d6-425f-a202-506d89080874-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.365977 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.365100    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.365977 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.365153    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.365977 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.365236    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.365977 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.365294    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0dea3a-53d6-425f-a202-506d89080874-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.367210 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.367178    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:44.367394 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.367375    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:44.367515 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.367500    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:44.370251 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.370235    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:54:44.370338 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.370277    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:54:44.370338 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.370298    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"]
Oct 30 19:54:44.370460 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.370345    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:54:44.370460 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.370382    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.370460 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.370397    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:54:44.370823 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.370802    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:54:44.412012 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.408557    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"]
Oct 30 19:54:44.466253 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.466217    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrw4b\" (UniqueName: \"kubernetes.io/projected/9d0dea3a-53d6-425f-a202-506d89080874-kube-api-access-lrw4b\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.466398 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.466289    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0dea3a-53d6-425f-a202-506d89080874-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.466398 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.466329    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.466398 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.466393    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.467255 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.466425    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.467255 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.466502    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0dea3a-53d6-425f-a202-506d89080874-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.467255 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.466529    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.467414 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.467262    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.467414 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.467323    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.467414 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.467369    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.467572 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.467414    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.467572 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.467456    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0dea3a-53d6-425f-a202-506d89080874-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.467572 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.467497    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.467572 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.467523    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.467788 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.467679    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.467847 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.467804    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.467920 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.467857    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.467920 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.467901    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m427\" (UniqueName: \"kubernetes.io/projected/86bda8bd-f59d-4f99-aafb-cccfbf84384c-kube-api-access-2m427\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.468899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.468838    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.470492 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.469629    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.470492 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.469760    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.470492 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.469911    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0dea3a-53d6-425f-a202-506d89080874-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.470492 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.470451    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0dea3a-53d6-425f-a202-506d89080874-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.470910 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.470887    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.471721 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.471685    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.488307 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.488246    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrw4b\" (UniqueName: \"kubernetes.io/projected/9d0dea3a-53d6-425f-a202-506d89080874-kube-api-access-lrw4b\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.489805 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.489781    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0dea3a-53d6-425f-a202-506d89080874-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.572883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.569853    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.572883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.569916    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m427\" (UniqueName: \"kubernetes.io/projected/86bda8bd-f59d-4f99-aafb-cccfbf84384c-kube-api-access-2m427\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.572883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.569995    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.572883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.570044    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.572883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.570078    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.572883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.570145    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.572883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.570209    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.572883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.570301    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.572883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.570326    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.572883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.571124    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.572883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.571706    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.572883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.571987    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.572883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.572634    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.572883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.572681    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.573722 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.573102    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.576170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.575947    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.607795 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.607712    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m427\" (UniqueName: \"kubernetes.io/projected/86bda8bd-f59d-4f99-aafb-cccfbf84384c-kube-api-access-2m427\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.607795 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.607781    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:44.662757 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.662031    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"]
Oct 30 19:54:44.665589 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.665558    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:44.666605 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:54:44.666573    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181fb3fa_5478_4c39_9cfe_de04f2070e49.slice/crio-9d8edc83d6a64380218b7bb3742c49815f85759d51c263a92ef572513a2c55bc WatchSource:0}: Error finding container 9d8edc83d6a64380218b7bb3742c49815f85759d51c263a92ef572513a2c55bc: Status 404 returned error can't find the container with id 9d8edc83d6a64380218b7bb3742c49815f85759d51c263a92ef572513a2c55bc
Oct 30 19:54:44.668813 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.668776    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:44.669229 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.668923    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:44.669229 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.668965    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:44.699929 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:44.699900    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:45.009754 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:54:45.009725    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0dea3a_53d6_425f_a202_506d89080874.slice/crio-68afaa5a6f8a40b22e796382b80a2a373a8b873241f01939cdc9a041374fe5e7 WatchSource:0}: Error finding container 68afaa5a6f8a40b22e796382b80a2a373a8b873241f01939cdc9a041374fe5e7: Status 404 returned error can't find the container with id 68afaa5a6f8a40b22e796382b80a2a373a8b873241f01939cdc9a041374fe5e7
Oct 30 19:54:45.010906 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.010879    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"]
Oct 30 19:54:45.012346 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.012307    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:45.012427 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.012368    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:45.012427 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.012399    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:45.039606 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.039577    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"]
Oct 30 19:54:45.048727 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:54:45.048694    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86bda8bd_f59d_4f99_aafb_cccfbf84384c.slice/crio-cc3352aa1a958e979fae7349a81e9c5a61601ffab1acbb5ef99c1c91b7298c38 WatchSource:0}: Error finding container cc3352aa1a958e979fae7349a81e9c5a61601ffab1acbb5ef99c1c91b7298c38: Status 404 returned error can't find the container with id cc3352aa1a958e979fae7349a81e9c5a61601ffab1acbb5ef99c1c91b7298c38
Oct 30 19:54:45.052090 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.052054    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:45.052185 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.052140    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:45.052256 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.052186    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:45.427339 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.427259    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps" event={"ID":"181fb3fa-5478-4c39-9cfe-de04f2070e49","Type":"ContainerStarted","Data":"502e0d62e64e8d721ab9505ac6578863ee10ee0c86be3c02f492c2df29d9b53c"}
Oct 30 19:54:45.427542 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.427421    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps" event={"ID":"181fb3fa-5478-4c39-9cfe-de04f2070e49","Type":"ContainerStarted","Data":"9d8edc83d6a64380218b7bb3742c49815f85759d51c263a92ef572513a2c55bc"}
Oct 30 19:54:45.435899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.435843    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf" event={"ID":"86bda8bd-f59d-4f99-aafb-cccfbf84384c","Type":"ContainerStarted","Data":"1308912743c5704062e89d13177e28a8948585cf59d0bf0dd0b843991fce079c"}
Oct 30 19:54:45.436142 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.436120    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf" event={"ID":"86bda8bd-f59d-4f99-aafb-cccfbf84384c","Type":"ContainerStarted","Data":"cc3352aa1a958e979fae7349a81e9c5a61601ffab1acbb5ef99c1c91b7298c38"}
Oct 30 19:54:45.443116 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.443086    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc" event={"ID":"9d0dea3a-53d6-425f-a202-506d89080874","Type":"ContainerStarted","Data":"ce6cc0cac78963b2a50d327839b01867b4c3c47af28467b06d55c5e251b38d31"}
Oct 30 19:54:45.443821 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.443796    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc" event={"ID":"9d0dea3a-53d6-425f-a202-506d89080874","Type":"ContainerStarted","Data":"68afaa5a6f8a40b22e796382b80a2a373a8b873241f01939cdc9a041374fe5e7"}
Oct 30 19:54:45.450567 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.450536    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n" event={"ID":"0947f714-d384-4e21-829a-cb2ab5f70636","Type":"ContainerStarted","Data":"8d49214b8251430b7e39f9e4d35fb7cb5b3683d8741605019815cf20fc5940e0"}
Oct 30 19:54:45.450678 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.450591    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n" event={"ID":"0947f714-d384-4e21-829a-cb2ab5f70636","Type":"ContainerStarted","Data":"b0dedbc364e62b0ccf1ea854179097f91aa1a0e4f65f9f415b147837c6170248"}
Oct 30 19:54:45.466982 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.466913    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps" podStartSLOduration=4.466899292 podStartE2EDuration="4.466899292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:45.464454516 +0000 UTC m=+1485.357068842" watchObservedRunningTime="2026-10-30 19:54:45.466899292 +0000 UTC m=+1485.359513609"
Oct 30 19:54:45.510279 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.509458    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf" podStartSLOduration=4.509438778 podStartE2EDuration="4.509438778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:45.502363023 +0000 UTC m=+1485.394977348" watchObservedRunningTime="2026-10-30 19:54:45.509438778 +0000 UTC m=+1485.402053103"
Oct 30 19:54:45.564461 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.564401    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc" podStartSLOduration=4.564383362 podStartE2EDuration="4.564383362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:45.555084165 +0000 UTC m=+1485.447698490" watchObservedRunningTime="2026-10-30 19:54:45.564383362 +0000 UTC m=+1485.456997689"
Oct 30 19:54:45.588113 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.587967    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n" podStartSLOduration=6.587947721 podStartE2EDuration="6.587947721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:45.586944905 +0000 UTC m=+1485.479559230" watchObservedRunningTime="2026-10-30 19:54:45.587947721 +0000 UTC m=+1485.480562042"
Oct 30 19:54:45.666884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.665965    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:45.689932 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.689664    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:45.697451 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.697416    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:45.704167 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:45.703746    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:46.444977 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:46.444910    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:54:46.445286 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:46.445246    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps" podUID="181fb3fa-5478-4c39-9cfe-de04f2070e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 19:54:46.747384 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:46.747274    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:54:46.747930 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:46.747377    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf" podUID="86bda8bd-f59d-4f99-aafb-cccfbf84384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 19:54:46.747930 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:46.747793    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:54:46.747930 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:46.747851    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc" podUID="9d0dea3a-53d6-425f-a202-506d89080874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 19:54:46.750319 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:46.750278    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:46.846943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:46.816692    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:46.847279 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:46.847257    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:54:47.354959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:47.353288    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:54:47.354959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:47.353370    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps" podUID="181fb3fa-5478-4c39-9cfe-de04f2070e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 19:54:47.722888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:47.722003    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:54:47.722888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:47.722082    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf" podUID="86bda8bd-f59d-4f99-aafb-cccfbf84384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 19:54:47.722888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:47.722535    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:54:47.722888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:47.722577    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc" podUID="9d0dea3a-53d6-425f-a202-506d89080874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 19:54:48.265952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:48.265251    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:54:48.265952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:48.265313    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps" podUID="181fb3fa-5478-4c39-9cfe-de04f2070e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 19:54:48.690907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:48.673783    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:54:48.690907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:48.673855    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc" podUID="9d0dea3a-53d6-425f-a202-506d89080874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 19:54:48.703491 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:48.703457    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:54:48.722887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:48.703729    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf" podUID="86bda8bd-f59d-4f99-aafb-cccfbf84384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 19:54:49.296907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:49.289186    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:54:49.296907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:49.289248    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps" podUID="181fb3fa-5478-4c39-9cfe-de04f2070e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 19:54:49.698049 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:49.698008    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:54:49.698322 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:49.698290    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc" podUID="9d0dea3a-53d6-425f-a202-506d89080874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 19:54:49.721232 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:49.721195    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:54:49.721486 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:49.721459    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf" podUID="86bda8bd-f59d-4f99-aafb-cccfbf84384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 19:54:50.245913 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.245873    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"]
Oct 30 19:54:50.307033 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.256690    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:50.352892 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.346435    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:54:50.353411 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.353367    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps" podUID="181fb3fa-5478-4c39-9cfe-de04f2070e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 19:54:50.381885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.379786    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:54:50.425896 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.425230    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:50.593943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.593682    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:54:50.840776 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.840710    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-workload-certs\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:50.861249 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.861212    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-workload-socket\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:50.861561 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.861542    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-envoy\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:50.861699 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.861679    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-credential-socket\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:50.861989 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.861968    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5706df-826d-4682-a77b-cbd1ae21bdef-istiod-ca-cert\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:50.863337 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.863309    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjlmf\" (UniqueName: \"kubernetes.io/projected/9e5706df-826d-4682-a77b-cbd1ae21bdef-kube-api-access-sjlmf\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:50.863494 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.863477    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-data\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:50.863631 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.863614    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-podinfo\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:50.863795 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.863779    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-token\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:50.863932 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.841979    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:54:50.864014 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.863987    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc" podUID="9d0dea3a-53d6-425f-a202-506d89080874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 19:54:50.864303 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.853727    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:50.881093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.881050    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:54:50.883404 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.883310    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf" podUID="86bda8bd-f59d-4f99-aafb-cccfbf84384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 19:54:51.003024 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:50.978359    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ea210528-openshift-default-dockercfg-8fw44\""
Oct 30 19:54:51.012301 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.006044    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-workload-certs\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.012590 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.012570    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-workload-socket\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.013174 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.013151    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-envoy\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.022383 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.022353    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-credential-socket\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.027927 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.022679    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5706df-826d-4682-a77b-cbd1ae21bdef-istiod-ca-cert\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.075287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.075250    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjlmf\" (UniqueName: \"kubernetes.io/projected/9e5706df-826d-4682-a77b-cbd1ae21bdef-kube-api-access-sjlmf\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.088458 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.088424    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-data\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.088704 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.088678    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-podinfo\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.089000 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.088977    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-token\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.096892 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.028487    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-credential-socket\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.343943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.007251    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"]
Oct 30 19:54:51.343943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.013035    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-workload-socket\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.343943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.097393    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-data\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.343943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.006431    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-workload-certs\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.343943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.029175    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5706df-826d-4682-a77b-cbd1ae21bdef-istiod-ca-cert\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.343943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.147753    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-podinfo\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.343943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.249970    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:51.343943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.279455    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:54:51.343943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.283100    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:51.343943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.286092    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:54:51.343943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.323579    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps" podUID="181fb3fa-5478-4c39-9cfe-de04f2070e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 19:54:51.343943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.324912    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:51.343943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.325237    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:54:51.434128 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.427661    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:51.435145 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.434591    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:54:51.451604 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.451546    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-token\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.540889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.477000    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-envoy\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.540889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.477107    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:51.540889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.478880    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded"
Oct 30 19:54:51.540889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.478999    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:54:51.720885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.714977    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:54:51.757584 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.725361    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:54:51.757584 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.725417    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc" podUID="9d0dea3a-53d6-425f-a202-506d89080874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 19:54:51.757584 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.725488    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:54:51.757584 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.725508    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf" podUID="86bda8bd-f59d-4f99-aafb-cccfbf84384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 19:54:51.843549 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.843505    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjlmf\" (UniqueName: \"kubernetes.io/projected/9e5706df-826d-4682-a77b-cbd1ae21bdef-kube-api-access-sjlmf\") pod \"e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") " pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.924888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.916159    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:51.947831 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:51.947798    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:54:52.277129 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:52.271770    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:54:52.277129 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:52.271828    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps" podUID="181fb3fa-5478-4c39-9cfe-de04f2070e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 19:54:52.365441 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:52.365410    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:54:52.666779 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:52.666556    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 30 19:54:52.666779 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:52.666618    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc" podUID="9d0dea3a-53d6-425f-a202-506d89080874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 30 19:54:52.700771 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:52.700721    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 19:54:52.701004 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:52.700789    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf" podUID="86bda8bd-f59d-4f99-aafb-cccfbf84384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 19:54:52.756166 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:52.756132    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"]
Oct 30 19:54:52.756300 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:52.756178    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:52.756300 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:52.756266    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:52.756428 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:52.756314    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:54:53.265781 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:53.265742    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:54:53.265986 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:53.265813    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps" podUID="181fb3fa-5478-4c39-9cfe-de04f2070e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Oct 30 19:54:53.561978 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:53.561884    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9" event={"ID":"9e5706df-826d-4682-a77b-cbd1ae21bdef","Type":"ContainerStarted","Data":"877d6784fc31f6d448670623b8d5e7adf39453fe5e7df6b2ad2f0d228b6181dd"}
Oct 30 19:54:53.562547 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:53.562524    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9" event={"ID":"9e5706df-826d-4682-a77b-cbd1ae21bdef","Type":"ContainerStarted","Data":"227bff5f0b7790a6685d5bb2f7bdfa6aec979df14683effa88de1a3513a7c514"}
Oct 30 19:54:53.611357 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:53.611287    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9" podStartSLOduration=4.611265462 podStartE2EDuration="4.611265462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:53.608389408 +0000 UTC m=+1493.501003749" watchObservedRunningTime="2026-10-30 19:54:53.611265462 +0000 UTC m=+1493.503879791"
Oct 30 19:54:53.997891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:53.997026    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:54.038890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:54.038517    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:54.262081 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:54.261993    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:54.264201 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:54.264181    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:54.268093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:54.268073    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 19:54:54.337108 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:54.337063    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:54.338988 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:54.337908    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:54.340079 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:54.340058    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 19:54:54.398431 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:54.398401    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:54.401513 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:54.399763    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:54.406060 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:54.406034    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 19:54:54.580318 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:54.579394    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:54.580786 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:54.580593    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:54:58.463130 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.463098    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"]
Oct 30 19:54:58.469175 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.469155    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"
Oct 30 19:54:58.475768 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.475530    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:54:58.475768 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.475709    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:54:58.476363 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.475805    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:54:58.476363 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.476076    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:54:58.480702 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.480676    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"]
Oct 30 19:54:58.624737 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.624698    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c8d01b-xpctk\" (UID: \"f20f22cc-d591-4eda-a2ed-5cb640ecd77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"
Oct 30 19:54:58.624913 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.624882    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c8d01b-xpctk\" (UID: \"f20f22cc-d591-4eda-a2ed-5cb640ecd77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"
Oct 30 19:54:58.624965 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.624945    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsd48\" (UniqueName: \"kubernetes.io/projected/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-kube-api-access-gsd48\") pod \"maas-api-revoke-sub-d7c8d01b-xpctk\" (UID: \"f20f22cc-d591-4eda-a2ed-5cb640ecd77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"
Oct 30 19:54:58.726575 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.726500    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c8d01b-xpctk\" (UID: \"f20f22cc-d591-4eda-a2ed-5cb640ecd77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"
Oct 30 19:54:58.726575 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.726561    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsd48\" (UniqueName: \"kubernetes.io/projected/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-kube-api-access-gsd48\") pod \"maas-api-revoke-sub-d7c8d01b-xpctk\" (UID: \"f20f22cc-d591-4eda-a2ed-5cb640ecd77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"
Oct 30 19:54:58.726768 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.726745    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c8d01b-xpctk\" (UID: \"f20f22cc-d591-4eda-a2ed-5cb640ecd77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"
Oct 30 19:54:58.727398 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.727369    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c8d01b-xpctk\" (UID: \"f20f22cc-d591-4eda-a2ed-5cb640ecd77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"
Oct 30 19:54:58.728873 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.728837    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c8d01b-xpctk\" (UID: \"f20f22cc-d591-4eda-a2ed-5cb640ecd77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"
Oct 30 19:54:58.739070 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.739048    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsd48\" (UniqueName: \"kubernetes.io/projected/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-kube-api-access-gsd48\") pod \"maas-api-revoke-sub-d7c8d01b-xpctk\" (UID: \"f20f22cc-d591-4eda-a2ed-5cb640ecd77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"
Oct 30 19:54:58.794501 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:58.794157    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"
Oct 30 19:54:59.190612 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:59.190574    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"]
Oct 30 19:54:59.483221 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:59.483140    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"]
Oct 30 19:54:59.489766 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:59.489738    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"
Oct 30 19:54:59.508659 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:59.508621    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"]
Oct 30 19:54:59.646958 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:59.646917    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714841b4-a1f6-4d03-9d2a-e1ab18481627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa8a8633-fb9rb\" (UID: \"714841b4-a1f6-4d03-9d2a-e1ab18481627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"
Oct 30 19:54:59.647223 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:59.646975    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdffh\" (UniqueName: \"kubernetes.io/projected/714841b4-a1f6-4d03-9d2a-e1ab18481627-kube-api-access-mdffh\") pod \"maas-api-revoke-sub-aa8a8633-fb9rb\" (UID: \"714841b4-a1f6-4d03-9d2a-e1ab18481627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"
Oct 30 19:54:59.647223 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:59.647011    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714841b4-a1f6-4d03-9d2a-e1ab18481627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa8a8633-fb9rb\" (UID: \"714841b4-a1f6-4d03-9d2a-e1ab18481627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"
Oct 30 19:54:59.677642 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:59.677538    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk" event={"ID":"f20f22cc-d591-4eda-a2ed-5cb640ecd77e","Type":"ContainerStarted","Data":"93120e8e4a836bdda2f8f7d33b7de64c9f0c2820d952bce005ad0c8ad9dffb56"}
Oct 30 19:54:59.749468 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:59.749375    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714841b4-a1f6-4d03-9d2a-e1ab18481627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa8a8633-fb9rb\" (UID: \"714841b4-a1f6-4d03-9d2a-e1ab18481627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"
Oct 30 19:54:59.749468 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:59.749438    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdffh\" (UniqueName: \"kubernetes.io/projected/714841b4-a1f6-4d03-9d2a-e1ab18481627-kube-api-access-mdffh\") pod \"maas-api-revoke-sub-aa8a8633-fb9rb\" (UID: \"714841b4-a1f6-4d03-9d2a-e1ab18481627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"
Oct 30 19:54:59.749667 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:59.749474    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714841b4-a1f6-4d03-9d2a-e1ab18481627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa8a8633-fb9rb\" (UID: \"714841b4-a1f6-4d03-9d2a-e1ab18481627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"
Oct 30 19:54:59.750759 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:59.750725    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714841b4-a1f6-4d03-9d2a-e1ab18481627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa8a8633-fb9rb\" (UID: \"714841b4-a1f6-4d03-9d2a-e1ab18481627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"
Oct 30 19:54:59.764914 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:59.753039    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714841b4-a1f6-4d03-9d2a-e1ab18481627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa8a8633-fb9rb\" (UID: \"714841b4-a1f6-4d03-9d2a-e1ab18481627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"
Oct 30 19:54:59.779965 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:59.779930    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdffh\" (UniqueName: \"kubernetes.io/projected/714841b4-a1f6-4d03-9d2a-e1ab18481627-kube-api-access-mdffh\") pod \"maas-api-revoke-sub-aa8a8633-fb9rb\" (UID: \"714841b4-a1f6-4d03-9d2a-e1ab18481627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"
Oct 30 19:54:59.809146 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:54:59.809113    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"
Oct 30 19:55:00.194498 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:55:00.194467    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714841b4_a1f6_4d03_9d2a_e1ab18481627.slice/crio-b17f74efa196b38c00813f6874d435bde0ec2442a0d89a29c29aad0ad4555351 WatchSource:0}: Error finding container b17f74efa196b38c00813f6874d435bde0ec2442a0d89a29c29aad0ad4555351: Status 404 returned error can't find the container with id b17f74efa196b38c00813f6874d435bde0ec2442a0d89a29c29aad0ad4555351
Oct 30 19:55:00.196292 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:00.196268    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"]
Oct 30 19:55:00.687277 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:00.687246    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 19:55:00.695770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:00.695721    4823 generic.go:356] "Generic (PLEG): container finished" podID="f20f22cc-d591-4eda-a2ed-5cb640ecd77e" containerID="0075beafc1fb515ecd335c7a58bf2d51b0d4660bf5b59f0e16028c18e19369cf" exitCode=0
Oct 30 19:55:00.696097 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:00.696067    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk" event={"ID":"f20f22cc-d591-4eda-a2ed-5cb640ecd77e","Type":"ContainerDied","Data":"0075beafc1fb515ecd335c7a58bf2d51b0d4660bf5b59f0e16028c18e19369cf"}
Oct 30 19:55:00.702094 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:00.702059    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb" event={"ID":"714841b4-a1f6-4d03-9d2a-e1ab18481627","Type":"ContainerStarted","Data":"694a9d22e525edb29b6a79cca847343673d4e16e7660bdb48d3e67356f6c1697"}
Oct 30 19:55:00.702259 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:00.702116    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb" event={"ID":"714841b4-a1f6-4d03-9d2a-e1ab18481627","Type":"ContainerStarted","Data":"b17f74efa196b38c00813f6874d435bde0ec2442a0d89a29c29aad0ad4555351"}
Oct 30 19:55:00.704276 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:00.704251    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 19:55:00.747539 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:00.747490    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb" podStartSLOduration=1.747476406 podStartE2EDuration="1.747476406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:00.743793493 +0000 UTC m=+1500.636407817" watchObservedRunningTime="2026-10-30 19:55:00.747476406 +0000 UTC m=+1500.640090729"
Oct 30 19:55:01.728911 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:01.720464    4823 generic.go:356] "Generic (PLEG): container finished" podID="714841b4-a1f6-4d03-9d2a-e1ab18481627" containerID="694a9d22e525edb29b6a79cca847343673d4e16e7660bdb48d3e67356f6c1697" exitCode=0
Oct 30 19:55:01.728911 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:01.720526    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb" event={"ID":"714841b4-a1f6-4d03-9d2a-e1ab18481627","Type":"ContainerDied","Data":"694a9d22e525edb29b6a79cca847343673d4e16e7660bdb48d3e67356f6c1697"}
Oct 30 19:55:01.949565 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:01.949481    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"
Oct 30 19:55:02.304975 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:02.241362    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-maas-api-openshift-ca\") pod \"f20f22cc-d591-4eda-a2ed-5cb640ecd77e\" (UID: \"f20f22cc-d591-4eda-a2ed-5cb640ecd77e\") "
Oct 30 19:55:02.304975 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:02.241423    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-kube-api-access-gsd48\" (UniqueName: \"kubernetes.io/projected/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-kube-api-access-gsd48\") pod \"f20f22cc-d591-4eda-a2ed-5cb640ecd77e\" (UID: \"f20f22cc-d591-4eda-a2ed-5cb640ecd77e\") "
Oct 30 19:55:02.304975 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:02.241574    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-maas-api-xks-ca\") pod \"f20f22cc-d591-4eda-a2ed-5cb640ecd77e\" (UID: \"f20f22cc-d591-4eda-a2ed-5cb640ecd77e\") "
Oct 30 19:55:02.304975 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:02.253076    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-maas-api-openshift-ca" pod "f20f22cc-d591-4eda-a2ed-5cb640ecd77e" (UID: "f20f22cc-d591-4eda-a2ed-5cb640ecd77e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:02.320086 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:02.320030    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-maas-api-xks-ca" pod "f20f22cc-d591-4eda-a2ed-5cb640ecd77e" (UID: "f20f22cc-d591-4eda-a2ed-5cb640ecd77e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:02.320423 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:02.320398    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-kube-api-access-gsd48" pod "f20f22cc-d591-4eda-a2ed-5cb640ecd77e" (UID: "f20f22cc-d591-4eda-a2ed-5cb640ecd77e"). InnerVolumeSpecName "kube-api-access-gsd48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:02.356420 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:55:02.356375    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:55:02.361156 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:02.360900    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:02.361156 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:02.360922    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:02.361156 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:02.360935    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsd48\" (UniqueName: \"kubernetes.io/projected/f20f22cc-d591-4eda-a2ed-5cb640ecd77e-kube-api-access-gsd48\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:02.761122 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:02.749317    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"
Oct 30 19:55:02.761122 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:02.749460    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk" event={"ID":"f20f22cc-d591-4eda-a2ed-5cb640ecd77e","Type":"ContainerDied","Data":"93120e8e4a836bdda2f8f7d33b7de64c9f0c2820d952bce005ad0c8ad9dffb56"}
Oct 30 19:55:02.761122 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:02.749536    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93120e8e4a836bdda2f8f7d33b7de64c9f0c2820d952bce005ad0c8ad9dffb56"
Oct 30 19:55:03.021041 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:03.020974    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"
Oct 30 19:55:03.105487 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:03.105446    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/714841b4-a1f6-4d03-9d2a-e1ab18481627-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714841b4-a1f6-4d03-9d2a-e1ab18481627-maas-api-xks-ca\") pod \"714841b4-a1f6-4d03-9d2a-e1ab18481627\" (UID: \"714841b4-a1f6-4d03-9d2a-e1ab18481627\") "
Oct 30 19:55:03.105675 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:03.105524    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714841b4-a1f6-4d03-9d2a-e1ab18481627-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714841b4-a1f6-4d03-9d2a-e1ab18481627-maas-api-openshift-ca\") pod \"714841b4-a1f6-4d03-9d2a-e1ab18481627\" (UID: \"714841b4-a1f6-4d03-9d2a-e1ab18481627\") "
Oct 30 19:55:03.105675 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:03.105651    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714841b4-a1f6-4d03-9d2a-e1ab18481627-kube-api-access-mdffh\" (UniqueName: \"kubernetes.io/projected/714841b4-a1f6-4d03-9d2a-e1ab18481627-kube-api-access-mdffh\") pod \"714841b4-a1f6-4d03-9d2a-e1ab18481627\" (UID: \"714841b4-a1f6-4d03-9d2a-e1ab18481627\") "
Oct 30 19:55:03.106074 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:03.106038    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714841b4-a1f6-4d03-9d2a-e1ab18481627-maas-api-openshift-ca" pod "714841b4-a1f6-4d03-9d2a-e1ab18481627" (UID: "714841b4-a1f6-4d03-9d2a-e1ab18481627"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:03.107212 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:03.107183    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714841b4-a1f6-4d03-9d2a-e1ab18481627-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:03.108281 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:03.108247    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714841b4-a1f6-4d03-9d2a-e1ab18481627-maas-api-xks-ca" pod "714841b4-a1f6-4d03-9d2a-e1ab18481627" (UID: "714841b4-a1f6-4d03-9d2a-e1ab18481627"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:03.108699 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:03.108673    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714841b4-a1f6-4d03-9d2a-e1ab18481627-kube-api-access-mdffh" pod "714841b4-a1f6-4d03-9d2a-e1ab18481627" (UID: "714841b4-a1f6-4d03-9d2a-e1ab18481627"). InnerVolumeSpecName "kube-api-access-mdffh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:03.208494 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:03.208454    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714841b4-a1f6-4d03-9d2a-e1ab18481627-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:03.208494 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:03.208488    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdffh\" (UniqueName: \"kubernetes.io/projected/714841b4-a1f6-4d03-9d2a-e1ab18481627-kube-api-access-mdffh\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:03.760456 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:03.760427    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"
Oct 30 19:55:03.760456 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:03.760442    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb" event={"ID":"714841b4-a1f6-4d03-9d2a-e1ab18481627","Type":"ContainerDied","Data":"b17f74efa196b38c00813f6874d435bde0ec2442a0d89a29c29aad0ad4555351"}
Oct 30 19:55:03.760654 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:03.760485    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b17f74efa196b38c00813f6874d435bde0ec2442a0d89a29c29aad0ad4555351"
Oct 30 19:55:04.319902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.306074    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"]
Oct 30 19:55:04.319902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.307121    4823 cpu_manager.go:414] "containerMap: removing container" podUID="f20f22cc-d591-4eda-a2ed-5cb640ecd77e" containerName="revoke-keys"
Oct 30 19:55:04.319902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.307141    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f20f22cc-d591-4eda-a2ed-5cb640ecd77e" containerName="revoke-keys"
Oct 30 19:55:04.319902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.307182    4823 cpu_manager.go:414] "containerMap: removing container" podUID="714841b4-a1f6-4d03-9d2a-e1ab18481627" containerName="revoke-keys"
Oct 30 19:55:04.319902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.307189    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="714841b4-a1f6-4d03-9d2a-e1ab18481627" containerName="revoke-keys"
Oct 30 19:55:04.319902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.307606    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="714841b4-a1f6-4d03-9d2a-e1ab18481627" containerName="revoke-keys"
Oct 30 19:55:04.319902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.307646    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="f20f22cc-d591-4eda-a2ed-5cb640ecd77e" containerName="revoke-keys"
Oct 30 19:55:04.343470 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.321392    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"
Oct 30 19:55:04.344060 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.344021    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:55:04.344169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.344104    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:55:04.344169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.344164    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:55:04.344725 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.344702    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"]
Oct 30 19:55:04.347571 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.347094    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:55:04.444948 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.444913    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv25c\" (UniqueName: \"kubernetes.io/projected/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-kube-api-access-pv25c\") pod \"maas-api-revoke-sub-7ce5193e-5gmg9\" (UID: \"6efb2f77-0732-43b8-80f4-65cc5ee19ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"
Oct 30 19:55:04.445149 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.444993    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce5193e-5gmg9\" (UID: \"6efb2f77-0732-43b8-80f4-65cc5ee19ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"
Oct 30 19:55:04.445149 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.445115    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce5193e-5gmg9\" (UID: \"6efb2f77-0732-43b8-80f4-65cc5ee19ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"
Oct 30 19:55:04.546559 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.546519    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce5193e-5gmg9\" (UID: \"6efb2f77-0732-43b8-80f4-65cc5ee19ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"
Oct 30 19:55:04.546795 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.546780    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv25c\" (UniqueName: \"kubernetes.io/projected/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-kube-api-access-pv25c\") pod \"maas-api-revoke-sub-7ce5193e-5gmg9\" (UID: \"6efb2f77-0732-43b8-80f4-65cc5ee19ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"
Oct 30 19:55:04.546924 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.546907    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce5193e-5gmg9\" (UID: \"6efb2f77-0732-43b8-80f4-65cc5ee19ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"
Oct 30 19:55:04.547321 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.547296    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce5193e-5gmg9\" (UID: \"6efb2f77-0732-43b8-80f4-65cc5ee19ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"
Oct 30 19:55:04.549242 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.549224    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce5193e-5gmg9\" (UID: \"6efb2f77-0732-43b8-80f4-65cc5ee19ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"
Oct 30 19:55:04.561234 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.561189    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv25c\" (UniqueName: \"kubernetes.io/projected/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-kube-api-access-pv25c\") pod \"maas-api-revoke-sub-7ce5193e-5gmg9\" (UID: \"6efb2f77-0732-43b8-80f4-65cc5ee19ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"
Oct 30 19:55:04.653226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:04.653195    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"
Oct 30 19:55:05.016329 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:55:05.016283    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6efb2f77_0732_43b8_80f4_65cc5ee19ec0.slice/crio-29d376be884154b968942e9791894dd2a7431dbbbe95301d4175405d0eac16ec WatchSource:0}: Error finding container 29d376be884154b968942e9791894dd2a7431dbbbe95301d4175405d0eac16ec: Status 404 returned error can't find the container with id 29d376be884154b968942e9791894dd2a7431dbbbe95301d4175405d0eac16ec
Oct 30 19:55:05.017715 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:05.017692    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"]
Oct 30 19:55:05.785120 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:05.785083    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9" event={"ID":"6efb2f77-0732-43b8-80f4-65cc5ee19ec0","Type":"ContainerStarted","Data":"ecaf5cebb29e670414a2431477d240305f2f7ef5de48d2784b5836309be37aa6"}
Oct 30 19:55:05.785757 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:05.785145    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9" event={"ID":"6efb2f77-0732-43b8-80f4-65cc5ee19ec0","Type":"ContainerStarted","Data":"29d376be884154b968942e9791894dd2a7431dbbbe95301d4175405d0eac16ec"}
Oct 30 19:55:05.824563 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:05.824511    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9" podStartSLOduration=1.8244959349999998 podStartE2EDuration="1.824495935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:05.820345519 +0000 UTC m=+1505.712959842" watchObservedRunningTime="2026-10-30 19:55:05.824495935 +0000 UTC m=+1505.717110259"
Oct 30 19:55:06.796768 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:06.796732    4823 generic.go:356] "Generic (PLEG): container finished" podID="6efb2f77-0732-43b8-80f4-65cc5ee19ec0" containerID="ecaf5cebb29e670414a2431477d240305f2f7ef5de48d2784b5836309be37aa6" exitCode=0
Oct 30 19:55:06.797217 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:06.796820    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9" event={"ID":"6efb2f77-0732-43b8-80f4-65cc5ee19ec0","Type":"ContainerDied","Data":"ecaf5cebb29e670414a2431477d240305f2f7ef5de48d2784b5836309be37aa6"}
Oct 30 19:55:08.050477 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.050454    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"
Oct 30 19:55:08.108443 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.108400    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-maas-api-openshift-ca\") pod \"6efb2f77-0732-43b8-80f4-65cc5ee19ec0\" (UID: \"6efb2f77-0732-43b8-80f4-65cc5ee19ec0\") "
Oct 30 19:55:08.108627 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.108607    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-maas-api-xks-ca\") pod \"6efb2f77-0732-43b8-80f4-65cc5ee19ec0\" (UID: \"6efb2f77-0732-43b8-80f4-65cc5ee19ec0\") "
Oct 30 19:55:08.108703 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.108685    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-kube-api-access-pv25c\" (UniqueName: \"kubernetes.io/projected/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-kube-api-access-pv25c\") pod \"6efb2f77-0732-43b8-80f4-65cc5ee19ec0\" (UID: \"6efb2f77-0732-43b8-80f4-65cc5ee19ec0\") "
Oct 30 19:55:08.108769 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.108738    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-maas-api-openshift-ca" pod "6efb2f77-0732-43b8-80f4-65cc5ee19ec0" (UID: "6efb2f77-0732-43b8-80f4-65cc5ee19ec0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:08.109407 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.109380    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:08.111336 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.111312    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-maas-api-xks-ca" pod "6efb2f77-0732-43b8-80f4-65cc5ee19ec0" (UID: "6efb2f77-0732-43b8-80f4-65cc5ee19ec0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:08.111443 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.111354    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-kube-api-access-pv25c" pod "6efb2f77-0732-43b8-80f4-65cc5ee19ec0" (UID: "6efb2f77-0732-43b8-80f4-65cc5ee19ec0"). InnerVolumeSpecName "kube-api-access-pv25c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:08.211092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.211047    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:08.211092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.211089    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv25c\" (UniqueName: \"kubernetes.io/projected/6efb2f77-0732-43b8-80f4-65cc5ee19ec0-kube-api-access-pv25c\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:08.816105 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.816077    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"
Oct 30 19:55:08.816292 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.816114    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9" event={"ID":"6efb2f77-0732-43b8-80f4-65cc5ee19ec0","Type":"ContainerDied","Data":"29d376be884154b968942e9791894dd2a7431dbbbe95301d4175405d0eac16ec"}
Oct 30 19:55:08.816292 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.816171    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29d376be884154b968942e9791894dd2a7431dbbbe95301d4175405d0eac16ec"
Oct 30 19:55:08.904797 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.904755    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"]
Oct 30 19:55:08.905609 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.905587    4823 cpu_manager.go:414] "containerMap: removing container" podUID="6efb2f77-0732-43b8-80f4-65cc5ee19ec0" containerName="revoke-keys"
Oct 30 19:55:08.905737 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.905610    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6efb2f77-0732-43b8-80f4-65cc5ee19ec0" containerName="revoke-keys"
Oct 30 19:55:08.905795 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.905783    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="6efb2f77-0732-43b8-80f4-65cc5ee19ec0" containerName="revoke-keys"
Oct 30 19:55:08.909941 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.909919    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"
Oct 30 19:55:08.916147 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.916111    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:55:08.916288 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.916167    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:55:08.916790 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.916765    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:55:08.917594 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.916799    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:55:08.927803 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:08.927776    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"]
Oct 30 19:55:09.023162 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.023130    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ed473f-9659-420e-af3d-0f7a6dcc2af6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34bfc24d-9nxbw\" (UID: \"76ed473f-9659-420e-af3d-0f7a6dcc2af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"
Oct 30 19:55:09.023332 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.023190    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r5sk\" (UniqueName: \"kubernetes.io/projected/76ed473f-9659-420e-af3d-0f7a6dcc2af6-kube-api-access-4r5sk\") pod \"maas-api-revoke-sub-34bfc24d-9nxbw\" (UID: \"76ed473f-9659-420e-af3d-0f7a6dcc2af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"
Oct 30 19:55:09.023429 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.023414    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ed473f-9659-420e-af3d-0f7a6dcc2af6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34bfc24d-9nxbw\" (UID: \"76ed473f-9659-420e-af3d-0f7a6dcc2af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"
Oct 30 19:55:09.125282 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.125252    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ed473f-9659-420e-af3d-0f7a6dcc2af6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34bfc24d-9nxbw\" (UID: \"76ed473f-9659-420e-af3d-0f7a6dcc2af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"
Oct 30 19:55:09.125717 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.125341    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ed473f-9659-420e-af3d-0f7a6dcc2af6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34bfc24d-9nxbw\" (UID: \"76ed473f-9659-420e-af3d-0f7a6dcc2af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"
Oct 30 19:55:09.125717 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.125402    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5sk\" (UniqueName: \"kubernetes.io/projected/76ed473f-9659-420e-af3d-0f7a6dcc2af6-kube-api-access-4r5sk\") pod \"maas-api-revoke-sub-34bfc24d-9nxbw\" (UID: \"76ed473f-9659-420e-af3d-0f7a6dcc2af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"
Oct 30 19:55:09.125956 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.125929    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ed473f-9659-420e-af3d-0f7a6dcc2af6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34bfc24d-9nxbw\" (UID: \"76ed473f-9659-420e-af3d-0f7a6dcc2af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"
Oct 30 19:55:09.127700 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.127674    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ed473f-9659-420e-af3d-0f7a6dcc2af6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34bfc24d-9nxbw\" (UID: \"76ed473f-9659-420e-af3d-0f7a6dcc2af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"
Oct 30 19:55:09.142743 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.142718    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5sk\" (UniqueName: \"kubernetes.io/projected/76ed473f-9659-420e-af3d-0f7a6dcc2af6-kube-api-access-4r5sk\") pod \"maas-api-revoke-sub-34bfc24d-9nxbw\" (UID: \"76ed473f-9659-420e-af3d-0f7a6dcc2af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"
Oct 30 19:55:09.229389 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.229354    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"
Oct 30 19:55:09.587502 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.582282    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"]
Oct 30 19:55:09.591517 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:55:09.591489    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ed473f_9659_420e_af3d_0f7a6dcc2af6.slice/crio-61655eb811ef225d37f57113709d663b7c392489c8aa7099e540668127b0d226 WatchSource:0}: Error finding container 61655eb811ef225d37f57113709d663b7c392489c8aa7099e540668127b0d226: Status 404 returned error can't find the container with id 61655eb811ef225d37f57113709d663b7c392489c8aa7099e540668127b0d226
Oct 30 19:55:09.826548 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.826508    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw" event={"ID":"76ed473f-9659-420e-af3d-0f7a6dcc2af6","Type":"ContainerStarted","Data":"61655eb811ef225d37f57113709d663b7c392489c8aa7099e540668127b0d226"}
Oct 30 19:55:09.865801 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.865770    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"]
Oct 30 19:55:09.870842 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.870822    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"
Oct 30 19:55:09.907580 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.907536    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"]
Oct 30 19:55:09.959958 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.958318    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23661b03-937a-4b72-b668-fb8dc0686ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49eeb928-nx7q6\" (UID: \"23661b03-937a-4b72-b668-fb8dc0686ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"
Oct 30 19:55:09.959958 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.958370    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwd2p\" (UniqueName: \"kubernetes.io/projected/23661b03-937a-4b72-b668-fb8dc0686ce3-kube-api-access-qwd2p\") pod \"maas-api-revoke-sub-49eeb928-nx7q6\" (UID: \"23661b03-937a-4b72-b668-fb8dc0686ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"
Oct 30 19:55:09.959958 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:09.958679    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23661b03-937a-4b72-b668-fb8dc0686ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49eeb928-nx7q6\" (UID: \"23661b03-937a-4b72-b668-fb8dc0686ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"
Oct 30 19:55:10.060679 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.060641    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23661b03-937a-4b72-b668-fb8dc0686ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49eeb928-nx7q6\" (UID: \"23661b03-937a-4b72-b668-fb8dc0686ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"
Oct 30 19:55:10.060679 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.060678    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwd2p\" (UniqueName: \"kubernetes.io/projected/23661b03-937a-4b72-b668-fb8dc0686ce3-kube-api-access-qwd2p\") pod \"maas-api-revoke-sub-49eeb928-nx7q6\" (UID: \"23661b03-937a-4b72-b668-fb8dc0686ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"
Oct 30 19:55:10.060971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.060853    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23661b03-937a-4b72-b668-fb8dc0686ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49eeb928-nx7q6\" (UID: \"23661b03-937a-4b72-b668-fb8dc0686ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"
Oct 30 19:55:10.061372 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.061350    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23661b03-937a-4b72-b668-fb8dc0686ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49eeb928-nx7q6\" (UID: \"23661b03-937a-4b72-b668-fb8dc0686ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"
Oct 30 19:55:10.063217 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.063195    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23661b03-937a-4b72-b668-fb8dc0686ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49eeb928-nx7q6\" (UID: \"23661b03-937a-4b72-b668-fb8dc0686ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"
Oct 30 19:55:10.073230 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.073206    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwd2p\" (UniqueName: \"kubernetes.io/projected/23661b03-937a-4b72-b668-fb8dc0686ce3-kube-api-access-qwd2p\") pod \"maas-api-revoke-sub-49eeb928-nx7q6\" (UID: \"23661b03-937a-4b72-b668-fb8dc0686ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"
Oct 30 19:55:10.186554 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.186478    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"
Oct 30 19:55:10.563104 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:55:10.563067    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23661b03_937a_4b72_b668_fb8dc0686ce3.slice/crio-bd7dde6465a00ce185bea77a55a97f4913ad9cc1c3581889f6802930e32929a7 WatchSource:0}: Error finding container bd7dde6465a00ce185bea77a55a97f4913ad9cc1c3581889f6802930e32929a7: Status 404 returned error can't find the container with id bd7dde6465a00ce185bea77a55a97f4913ad9cc1c3581889f6802930e32929a7
Oct 30 19:55:10.563571 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.563549    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"]
Oct 30 19:55:10.587898 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.587855    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"]
Oct 30 19:55:10.592817 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.592798    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.615481 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.615451    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"]
Oct 30 19:55:10.685191 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.685158    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.685338 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.685197    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.685414 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.685386    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.685462 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.685450    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.685576 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.685545    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.685611 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.685598    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.685662 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.685649    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.685697 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.685683    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnnz7\" (UniqueName: \"kubernetes.io/projected/ee4bdb13-b5bb-4692-bbff-769a263dcffa-kube-api-access-jnnz7\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.685805 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.685787    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.696917 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.696884    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"]
Oct 30 19:55:10.703140 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.703117    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.726789 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.726760    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"]
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.792748    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.792791    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.792811    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.792878    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b64h6\" (UniqueName: \"kubernetes.io/projected/6adff78e-1673-4bee-98b9-71771700b23f-kube-api-access-b64h6\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.792942    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.792982    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.793020    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6adff78e-1673-4bee-98b9-71771700b23f-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.793042    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6adff78e-1673-4bee-98b9-71771700b23f-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.793089    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.793115    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.793150    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.793191    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.793219    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnnz7\" (UniqueName: \"kubernetes.io/projected/ee4bdb13-b5bb-4692-bbff-769a263dcffa-kube-api-access-jnnz7\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.793303    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6adff78e-1673-4bee-98b9-71771700b23f-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.793338    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.793405    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.793575    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.793631    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.794287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.794147    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.796238 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.796194    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.796918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.796882    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.797397 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.797361    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.799469 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.799448    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.847255 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.847166    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6" event={"ID":"23661b03-937a-4b72-b668-fb8dc0686ce3","Type":"ContainerStarted","Data":"bd7dde6465a00ce185bea77a55a97f4913ad9cc1c3581889f6802930e32929a7"}
Oct 30 19:55:10.849090 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.849063    4823 generic.go:356] "Generic (PLEG): container finished" podID="76ed473f-9659-420e-af3d-0f7a6dcc2af6" containerID="e7d783ece075943f726e56b83e53e126f2f6a843fcbd42627c1a646435bf8530" exitCode=0
Oct 30 19:55:10.849198 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.849125    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw" event={"ID":"76ed473f-9659-420e-af3d-0f7a6dcc2af6","Type":"ContainerDied","Data":"e7d783ece075943f726e56b83e53e126f2f6a843fcbd42627c1a646435bf8530"}
Oct 30 19:55:10.853811 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.853770    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.854029 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.853999    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.854228 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.854210    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnnz7\" (UniqueName: \"kubernetes.io/projected/ee4bdb13-b5bb-4692-bbff-769a263dcffa-kube-api-access-jnnz7\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.854556 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.854526    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:10.895939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.895776    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.895939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.895818    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.895939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.895914    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.915396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.895965    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b64h6\" (UniqueName: \"kubernetes.io/projected/6adff78e-1673-4bee-98b9-71771700b23f-kube-api-access-b64h6\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.915396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.896026    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.915396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.896068    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6adff78e-1673-4bee-98b9-71771700b23f-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.915396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.896090    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6adff78e-1673-4bee-98b9-71771700b23f-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.915396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.896211    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6adff78e-1673-4bee-98b9-71771700b23f-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.915396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.896237    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.915396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.896267    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.915396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.896299    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.915396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.896409    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.915396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.896571    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.915396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.897028    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6adff78e-1673-4bee-98b9-71771700b23f-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.915396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.898540    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.915396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.898770    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6adff78e-1673-4bee-98b9-71771700b23f-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.915396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.911925    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6adff78e-1673-4bee-98b9-71771700b23f-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.915396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.912012    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b64h6\" (UniqueName: \"kubernetes.io/projected/6adff78e-1673-4bee-98b9-71771700b23f-kube-api-access-b64h6\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:10.972103 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:10.972068    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:11.029134 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.028986    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:11.274531 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:55:11.274500    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4bdb13_b5bb_4692_bbff_769a263dcffa.slice/crio-b043c5be1c925c66107117c2b5d8d8a5ee152d3c4a98a9dc7d79d0682210349b WatchSource:0}: Error finding container b043c5be1c925c66107117c2b5d8d8a5ee152d3c4a98a9dc7d79d0682210349b: Status 404 returned error can't find the container with id b043c5be1c925c66107117c2b5d8d8a5ee152d3c4a98a9dc7d79d0682210349b
Oct 30 19:55:11.276224 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.276202    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"]
Oct 30 19:55:11.277473 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.277439    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:55:11.277550 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.277500    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:55:11.277592 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.277567    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:55:11.354772 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.348711    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"]
Oct 30 19:55:11.356035 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.356010    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.372136 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:55:11.372102    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6adff78e_1673_4bee_98b9_71771700b23f.slice/crio-70d838ad0c7217853e6bf3fccc7d469029dc7cc2b15edcc30fc28949847d66c2 WatchSource:0}: Error finding container 70d838ad0c7217853e6bf3fccc7d469029dc7cc2b15edcc30fc28949847d66c2: Status 404 returned error can't find the container with id 70d838ad0c7217853e6bf3fccc7d469029dc7cc2b15edcc30fc28949847d66c2
Oct 30 19:55:11.377193 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.377165    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"]
Oct 30 19:55:11.387350 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.386992    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"]
Oct 30 19:55:11.391785 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.391752    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:55:11.391973 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.391814    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:55:11.391973 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.391842    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:55:11.522541 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.522504    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.522541 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.522547    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltdsm\" (UniqueName: \"kubernetes.io/projected/330c59cb-17b4-4a1b-8023-c98da7575a34-kube-api-access-ltdsm\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.522730 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.522636    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.522730 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.522664    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.522795 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.522736    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.522829 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.522817    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330c59cb-17b4-4a1b-8023-c98da7575a34-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.522964 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.522937    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.523150 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.523132    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.523200 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.523186    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.625472 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.625379    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.625472 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.625428    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.625770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.625484    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.625770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.625576    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330c59cb-17b4-4a1b-8023-c98da7575a34-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.625770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.625614    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.625770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.625731    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.625770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.625765    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.626124 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.625935    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.626124 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.625956    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdsm\" (UniqueName: \"kubernetes.io/projected/330c59cb-17b4-4a1b-8023-c98da7575a34-kube-api-access-ltdsm\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.626329 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.626302    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.626554 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.626531    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330c59cb-17b4-4a1b-8023-c98da7575a34-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.626652 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.626620    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.626853 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.626834    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.627114 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.627097    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.628520 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.628494    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.628928 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.628910    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.638050 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.638022    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.638239 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.638217    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltdsm\" (UniqueName: \"kubernetes.io/projected/330c59cb-17b4-4a1b-8023-c98da7575a34-kube-api-access-ltdsm\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.677662 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.677616    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:11.912762 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.912669    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6" event={"ID":"23661b03-937a-4b72-b668-fb8dc0686ce3","Type":"ContainerStarted","Data":"42fe5f7dfa46a2f7095fc5570aa73f8e227730f470bb967e9ad37f9742a10c3c"}
Oct 30 19:55:11.916135 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.916102    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" event={"ID":"6adff78e-1673-4bee-98b9-71771700b23f","Type":"ContainerStarted","Data":"b68f42c20d5aa0671e1cde35cab2bc0268b4f972805f2c3823a7c9a51c637f3e"}
Oct 30 19:55:11.916258 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.916160    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" event={"ID":"6adff78e-1673-4bee-98b9-71771700b23f","Type":"ContainerStarted","Data":"70d838ad0c7217853e6bf3fccc7d469029dc7cc2b15edcc30fc28949847d66c2"}
Oct 30 19:55:11.920796 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.920767    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" event={"ID":"ee4bdb13-b5bb-4692-bbff-769a263dcffa","Type":"ContainerStarted","Data":"05f06ff70606adbb5a5c1bd86284a755dade95ec61dee2c35e6a144fae263597"}
Oct 30 19:55:11.922493 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:11.922462    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" event={"ID":"ee4bdb13-b5bb-4692-bbff-769a263dcffa","Type":"ContainerStarted","Data":"b043c5be1c925c66107117c2b5d8d8a5ee152d3c4a98a9dc7d79d0682210349b"}
Oct 30 19:55:12.017081 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.001975    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:12.043771 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.042521    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:12.043771 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.026977    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podStartSLOduration=2.026851401 podStartE2EDuration="2.026851401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:11.962145089 +0000 UTC m=+1511.854759424" watchObservedRunningTime="2026-10-30 19:55:12.026851401 +0000 UTC m=+1511.919465804"
Oct 30 19:55:12.051635 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.014154    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:55:12.051635 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.044577    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:55:12.051635 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.039535    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:55:12.052804 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.052320    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:55:12.073768 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.073717    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:55:12.073997 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.073817    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:55:12.073997 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.073857    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:55:12.076704 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.076648    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podStartSLOduration=2.076631824 podStartE2EDuration="2.076631824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:12.044515065 +0000 UTC m=+1511.937129388" watchObservedRunningTime="2026-10-30 19:55:12.076631824 +0000 UTC m=+1511.969246150"
Oct 30 19:55:12.078208 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.078186    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"]
Oct 30 19:55:12.232072 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.232037    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"
Oct 30 19:55:12.349886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.349544    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76ed473f-9659-420e-af3d-0f7a6dcc2af6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ed473f-9659-420e-af3d-0f7a6dcc2af6-maas-api-xks-ca\") pod \"76ed473f-9659-420e-af3d-0f7a6dcc2af6\" (UID: \"76ed473f-9659-420e-af3d-0f7a6dcc2af6\") "
Oct 30 19:55:12.349886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.349799    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ed473f-9659-420e-af3d-0f7a6dcc2af6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ed473f-9659-420e-af3d-0f7a6dcc2af6-maas-api-openshift-ca\") pod \"76ed473f-9659-420e-af3d-0f7a6dcc2af6\" (UID: \"76ed473f-9659-420e-af3d-0f7a6dcc2af6\") "
Oct 30 19:55:12.374336 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.349914    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ed473f-9659-420e-af3d-0f7a6dcc2af6-kube-api-access-4r5sk\" (UniqueName: \"kubernetes.io/projected/76ed473f-9659-420e-af3d-0f7a6dcc2af6-kube-api-access-4r5sk\") pod \"76ed473f-9659-420e-af3d-0f7a6dcc2af6\" (UID: \"76ed473f-9659-420e-af3d-0f7a6dcc2af6\") "
Oct 30 19:55:12.374336 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.351470    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ed473f-9659-420e-af3d-0f7a6dcc2af6-maas-api-openshift-ca" pod "76ed473f-9659-420e-af3d-0f7a6dcc2af6" (UID: "76ed473f-9659-420e-af3d-0f7a6dcc2af6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:12.374336 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.386708    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ed473f-9659-420e-af3d-0f7a6dcc2af6-maas-api-xks-ca" pod "76ed473f-9659-420e-af3d-0f7a6dcc2af6" (UID: "76ed473f-9659-420e-af3d-0f7a6dcc2af6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:12.413550 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.413457    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ed473f-9659-420e-af3d-0f7a6dcc2af6-kube-api-access-4r5sk" pod "76ed473f-9659-420e-af3d-0f7a6dcc2af6" (UID: "76ed473f-9659-420e-af3d-0f7a6dcc2af6"). InnerVolumeSpecName "kube-api-access-4r5sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:12.455944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.452232    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ed473f-9659-420e-af3d-0f7a6dcc2af6-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:12.455944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.452266    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r5sk\" (UniqueName: \"kubernetes.io/projected/76ed473f-9659-420e-af3d-0f7a6dcc2af6-kube-api-access-4r5sk\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:12.455944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.452284    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ed473f-9659-420e-af3d-0f7a6dcc2af6-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:12.932096 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.932057    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" event={"ID":"330c59cb-17b4-4a1b-8023-c98da7575a34","Type":"ContainerStarted","Data":"0ee35e888bf11d1a4662ccac978efd213b6fb3fd02e4f2a27a7b99ce687e8fad"}
Oct 30 19:55:12.932287 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.932123    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" event={"ID":"330c59cb-17b4-4a1b-8023-c98da7575a34","Type":"ContainerStarted","Data":"54bc2afc4187114a017af79bbb3d90cddfb2ea3931b2116f67ad1747f8b423e0"}
Oct 30 19:55:12.933740 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.933713    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"
Oct 30 19:55:12.934025 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.933736    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw" event={"ID":"76ed473f-9659-420e-af3d-0f7a6dcc2af6","Type":"ContainerDied","Data":"61655eb811ef225d37f57113709d663b7c392489c8aa7099e540668127b0d226"}
Oct 30 19:55:12.934109 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.934052    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61655eb811ef225d37f57113709d663b7c392489c8aa7099e540668127b0d226"
Oct 30 19:55:12.935688 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.935664    4823 generic.go:356] "Generic (PLEG): container finished" podID="23661b03-937a-4b72-b668-fb8dc0686ce3" containerID="42fe5f7dfa46a2f7095fc5570aa73f8e227730f470bb967e9ad37f9742a10c3c" exitCode=0
Oct 30 19:55:12.935788 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:12.935742    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6" event={"ID":"23661b03-937a-4b72-b668-fb8dc0686ce3","Type":"ContainerDied","Data":"42fe5f7dfa46a2f7095fc5570aa73f8e227730f470bb967e9ad37f9742a10c3c"}
Oct 30 19:55:13.002964 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:13.001261    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podStartSLOduration=2.001239955 podStartE2EDuration="2.001239955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:12.999232719 +0000 UTC m=+1512.891847071" watchObservedRunningTime="2026-10-30 19:55:13.001239955 +0000 UTC m=+1512.893854281"
Oct 30 19:55:13.015004 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:13.004266    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:55:13.015289 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:13.015259    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:55:13.029801 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:13.029768    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:55:13.030224 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:13.030188    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:55:13.640665 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:13.635276    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"
Oct 30 19:55:13.683782 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:13.683741    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:13.695907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:13.691194    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:55:13.695907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:13.691254    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:55:13.853935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:13.828100    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23661b03-937a-4b72-b668-fb8dc0686ce3-kube-api-access-qwd2p\" (UniqueName: \"kubernetes.io/projected/23661b03-937a-4b72-b668-fb8dc0686ce3-kube-api-access-qwd2p\") pod \"23661b03-937a-4b72-b668-fb8dc0686ce3\" (UID: \"23661b03-937a-4b72-b668-fb8dc0686ce3\") "
Oct 30 19:55:13.853935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:13.828188    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23661b03-937a-4b72-b668-fb8dc0686ce3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23661b03-937a-4b72-b668-fb8dc0686ce3-maas-api-openshift-ca\") pod \"23661b03-937a-4b72-b668-fb8dc0686ce3\" (UID: \"23661b03-937a-4b72-b668-fb8dc0686ce3\") "
Oct 30 19:55:13.853935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:13.828320    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23661b03-937a-4b72-b668-fb8dc0686ce3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23661b03-937a-4b72-b668-fb8dc0686ce3-maas-api-xks-ca\") pod \"23661b03-937a-4b72-b668-fb8dc0686ce3\" (UID: \"23661b03-937a-4b72-b668-fb8dc0686ce3\") "
Oct 30 19:55:13.853935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:13.841968    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23661b03-937a-4b72-b668-fb8dc0686ce3-maas-api-openshift-ca" pod "23661b03-937a-4b72-b668-fb8dc0686ce3" (UID: "23661b03-937a-4b72-b668-fb8dc0686ce3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:13.853935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:13.845419    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23661b03-937a-4b72-b668-fb8dc0686ce3-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:13.853935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:13.850113    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23661b03-937a-4b72-b668-fb8dc0686ce3-maas-api-xks-ca" pod "23661b03-937a-4b72-b668-fb8dc0686ce3" (UID: "23661b03-937a-4b72-b668-fb8dc0686ce3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:14.005958 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:13.950835    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23661b03-937a-4b72-b668-fb8dc0686ce3-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:14.053267 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:14.050834    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:55:14.053267 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:14.050913    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:55:14.167224 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:14.153217    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:55:14.167224 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:14.153293    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:55:14.201731 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:14.201690    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23661b03-937a-4b72-b668-fb8dc0686ce3-kube-api-access-qwd2p" pod "23661b03-937a-4b72-b668-fb8dc0686ce3" (UID: "23661b03-937a-4b72-b668-fb8dc0686ce3"). InnerVolumeSpecName "kube-api-access-qwd2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:14.213012 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:14.212976    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"
Oct 30 19:55:14.220192 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:14.214188    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6" event={"ID":"23661b03-937a-4b72-b668-fb8dc0686ce3","Type":"ContainerDied","Data":"bd7dde6465a00ce185bea77a55a97f4913ad9cc1c3581889f6802930e32929a7"}
Oct 30 19:55:14.220192 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:14.219047    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd7dde6465a00ce185bea77a55a97f4913ad9cc1c3581889f6802930e32929a7"
Oct 30 19:55:14.320896 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:14.318817    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwd2p\" (UniqueName: \"kubernetes.io/projected/23661b03-937a-4b72-b668-fb8dc0686ce3-kube-api-access-qwd2p\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:14.790945 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:14.765498    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:55:15.021021 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.020971    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:55:15.148888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:14.985826    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:55:15.149186 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.149146    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:55:15.149646 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.051788    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:55:15.149854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.149827    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:55:15.174633 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.174597    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"]
Oct 30 19:55:15.252169 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.252126    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:15.310900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.259119    4823 cpu_manager.go:414] "containerMap: removing container" podUID="76ed473f-9659-420e-af3d-0f7a6dcc2af6" containerName="revoke-keys"
Oct 30 19:55:15.310900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.309068    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ed473f-9659-420e-af3d-0f7a6dcc2af6" containerName="revoke-keys"
Oct 30 19:55:15.310900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.309120    4823 cpu_manager.go:414] "containerMap: removing container" podUID="23661b03-937a-4b72-b668-fb8dc0686ce3" containerName="revoke-keys"
Oct 30 19:55:15.310900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.309129    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23661b03-937a-4b72-b668-fb8dc0686ce3" containerName="revoke-keys"
Oct 30 19:55:15.310900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.309424    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ed473f-9659-420e-af3d-0f7a6dcc2af6" containerName="revoke-keys"
Oct 30 19:55:15.310900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.309451    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="23661b03-937a-4b72-b668-fb8dc0686ce3" containerName="revoke-keys"
Oct 30 19:55:15.377178 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.377128    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:15.378872 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.378819    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:55:15.587907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.587780    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:15.635930 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.625196    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"
Oct 30 19:55:15.717348 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.702067    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:15.718770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.718735    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:55:15.719027 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.719000    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:55:15.759150 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.753003    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:15.759150 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.756141    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:55:15.759150 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.756659    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:55:15.769892 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.764938    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:55:15.783758 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.783240    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:55:15.953214 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.909215    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvssk\" (UniqueName: \"kubernetes.io/projected/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-kube-api-access-jvssk\") pod \"maas-api-revoke-sub-2f70995d-2p867\" (UID: \"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"
Oct 30 19:55:15.953214 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.909410    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f70995d-2p867\" (UID: \"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"
Oct 30 19:55:15.953214 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.909730    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f70995d-2p867\" (UID: \"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"
Oct 30 19:55:15.982885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.977699    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:55:15.990885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.989125    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:55:15.990885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:15.990255    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"]
Oct 30 19:55:16.015139 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.015096    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f70995d-2p867\" (UID: \"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"
Oct 30 19:55:16.078884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.066229    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvssk\" (UniqueName: \"kubernetes.io/projected/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-kube-api-access-jvssk\") pod \"maas-api-revoke-sub-2f70995d-2p867\" (UID: \"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"
Oct 30 19:55:16.078884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.066450    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f70995d-2p867\" (UID: \"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"
Oct 30 19:55:16.078884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.049342    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:55:16.078884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.067031    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:55:16.080047 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.080018    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f70995d-2p867\" (UID: \"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"
Oct 30 19:55:16.110900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.085382    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f70995d-2p867\" (UID: \"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"
Oct 30 19:55:16.140467 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.140309    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvssk\" (UniqueName: \"kubernetes.io/projected/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-kube-api-access-jvssk\") pod \"maas-api-revoke-sub-2f70995d-2p867\" (UID: \"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"
Oct 30 19:55:16.257885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.248956    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:16.259931 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.258163    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:55:16.312229 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.284729    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"
Oct 30 19:55:16.387047 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.387006    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:16.387383 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.387359    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:55:16.694886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.686319    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:55:16.694886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.686385    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:55:16.756941 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.755595    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:16.756941 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.756289    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:55:16.806196 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.806143    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:16.970190 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:16.970127    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:17.045737 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:17.045642    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:55:17.046040 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:17.045996    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:55:17.140073 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:17.140036    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:55:17.140352 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:17.140324    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:55:17.256034 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:17.254601    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:17.268795 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:17.267404    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:55:17.269593 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:17.269566    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:55:17.349281 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:17.348964    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:17.372343 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:17.372298    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:55:17.412469 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:17.412427    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:55:17.534291 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:17.497663    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:17.711130 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:17.711091    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:55:17.711428 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:17.711395    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:55:17.758030 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:17.757973    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:17.996265 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:17.995989    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:18.028882 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:18.010600    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:55:18.043882 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:18.035265    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:55:18.043882 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:18.035377    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:55:18.090888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:18.072163    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:55:18.117018 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:18.116959    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:55:18.334051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:18.229925    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 19:55:18.334051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:18.230361    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:55:18.334051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:18.258224    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:55:18.334051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:18.259018    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:55:18.334051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:18.275945    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:18.396022 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:18.395975    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:18.396554 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:18.396516    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:18.653887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:18.621526    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:55:18.653887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:18.627438    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:55:18.945037 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:18.938410    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:55:18.945037 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:18.938471    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:55:19.106149 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.106092    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:55:19.220952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.119230    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Oct 30 19:55:19.220952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.130193    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:19.339886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.336135    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:55:19.339886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.337812    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:55:19.377945 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.372287    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:55:19.377945 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.372620    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:55:19.377945 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.372674    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:55:19.545346 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.545221    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:55:19.636898 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.568202    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:55:19.638971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.638923    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:19.719236 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.719186    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:19.721001 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.720967    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:55:19.729073 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.729032    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:55:19.777902 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:55:19.777846    4823 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.476s"
Oct 30 19:55:19.912093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.861662    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:55:19.912093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.862014    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:55:19.912093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.862087    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:55:19.912093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.862160    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:55:19.912093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.779913    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 30 19:55:19.912093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:19.829146    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:20.641327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.070676    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:55:20.641327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.494054    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:55:20.641327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.179466    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:55:20.641327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.494232    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:55:20.641327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.201356    4823 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:20.641327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.494329    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm" podUID="ad11ffea-d313-4369-81f8-70014b50603f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:20.641327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.315009    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:55:20.641327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.494503    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:55:20.641327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.515438    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:55:20.641327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.522984    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:55:20.641327 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:55:20.576206    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f10e1fd_8a47_4d0a_8eb0_85d6763c7515.slice/crio-7c5ab0c7bde578bb8804e0ec70f3cb7f6e7955f2f3554fc82c0f57289c4469d5 WatchSource:0}: Error finding container 7c5ab0c7bde578bb8804e0ec70f3cb7f6e7955f2f3554fc82c0f57289c4469d5: Status 404 returned error can't find the container with id 7c5ab0c7bde578bb8804e0ec70f3cb7f6e7955f2f3554fc82c0f57289c4469d5
Oct 30 19:55:20.641327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.580239    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:55:20.675900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.654122    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:20.843890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.843459    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:55:20.851891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.844929    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:20.876914 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.868935    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:55:20.902981 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.889928    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:55:20.903686 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.903644    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:55:20.932061 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.932028    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:55:20.932339 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:20.932300    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:55:21.047384 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:21.047348    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:55:21.047694 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:21.047644    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:55:21.081428 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:21.081372    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:55:21.081757 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:21.081705    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:55:21.536763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:21.536717    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:21.537315 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:21.537283    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:55:21.539091 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:21.539059    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:21.899567 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:21.581946    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:21.961996 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:21.672508    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:21.986943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:21.839788    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:55:21.987379 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:21.987334    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:55:21.987614 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:21.847286    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:21.987747 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:21.870976    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:21.988003 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:21.926032    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:21.988154 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:21.958153    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:55:21.988463 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:21.845159    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:22.164941 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:55:22.142198    4823 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.555s"
Oct 30 19:55:22.234015 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:22.182417    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"]
Oct 30 19:55:22.234015 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:22.182724    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867" event={"ID":"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515","Type":"ContainerStarted","Data":"7c5ab0c7bde578bb8804e0ec70f3cb7f6e7955f2f3554fc82c0f57289c4469d5"}
Oct 30 19:55:22.308883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:22.283763    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:55:22.359890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:22.309150    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:55:22.469102 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:22.469011    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:55:22.469358 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:22.469329    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:55:22.540890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:22.537962    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:22.540890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:22.540073    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:55:22.728286 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:22.728191    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:55:22.731982 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:22.731934    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:55:22.838919 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:22.809917    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:23.010942 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:23.007471    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:55:23.036028 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:23.035985    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:55:23.036296 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:23.030231    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:55:23.036432 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:23.036410    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:55:23.148014 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:23.147964    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:23.149523 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:23.149491    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:55:23.150269 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:23.150242    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:55:23.193023 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:23.192969    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:23.226412 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:23.226368    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:23.256924 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:23.247648    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:55:23.257341 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:23.247683    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:55:23.560813 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:23.557714    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:23.662885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:23.659610    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:55:23.737887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:23.571000    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:23.799887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:23.765987    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:55:23.969300 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:23.969250    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:55:24.098485 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:24.093133    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:55:24.098485 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:24.093212    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:55:24.098485 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:24.093474    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:55:24.098485 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:24.093597    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:55:24.156607 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:24.156245    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:24.204321 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:24.204279    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:55:24.384229 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:24.264521    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:24.427876 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:24.414176    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:55:24.427876 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:24.425164    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:55:24.523637 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:24.523589    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:55:24.737943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:24.264141    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:55:24.737943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:24.695475    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:24.737943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:24.695752    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:24.747392 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:24.740628    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:24.794895 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:24.741106    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:55:24.794895 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:24.761998    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:55:25.049999 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.040983    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:55:25.074999 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.074949    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:55:25.075245 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.041019    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:55:25.075411 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.075387    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:55:25.177887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.177592    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:55:25.177887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.177634    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867" event={"ID":"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515","Type":"ContainerStarted","Data":"86139569599fb137c7e8e6a9225968db289ddc573f9de8674a5e34c98db59a4b"}
Oct 30 19:55:25.201912 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.194071    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:25.359950 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.335406    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:25.371968 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.361017    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:25.414958 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.414913    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"]
Oct 30 19:55:25.619947 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.596882    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:55:25.620680 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.620636    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:55:25.841825 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.840703    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"
Oct 30 19:55:25.921073 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.920967    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:25.938285 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.938240    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:55:25.998917 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.989806    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:55:25.998917 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.990117    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:55:25.998917 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.990368    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:25.998917 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.993512    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:55:25.998917 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:25.993565    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:55:26.000123 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.000092    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:55:26.000232 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.000168    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:55:26.042612 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.042568    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:26.048886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.045370    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:55:26.048886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.045424    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:55:26.201653 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.201430    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:55:26.249334 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.249101    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:26.351637 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.351603    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wv8f\" (UniqueName: \"kubernetes.io/projected/09d56c80-79bf-4e01-968f-3beab184830f-kube-api-access-6wv8f\") pod \"maas-api-revoke-sub-f40b9632-9nhn9\" (UID: \"09d56c80-79bf-4e01-968f-3beab184830f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"
Oct 30 19:55:26.362442 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.362157    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:26.370908 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.368839    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d56c80-79bf-4e01-968f-3beab184830f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f40b9632-9nhn9\" (UID: \"09d56c80-79bf-4e01-968f-3beab184830f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"
Oct 30 19:55:26.370908 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.369314    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d56c80-79bf-4e01-968f-3beab184830f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f40b9632-9nhn9\" (UID: \"09d56c80-79bf-4e01-968f-3beab184830f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"
Oct 30 19:55:26.608801 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.608769    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"]
Oct 30 19:55:26.620887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.608938    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d56c80-79bf-4e01-968f-3beab184830f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f40b9632-9nhn9\" (UID: \"09d56c80-79bf-4e01-968f-3beab184830f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"
Oct 30 19:55:26.637594 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.637556    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d56c80-79bf-4e01-968f-3beab184830f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f40b9632-9nhn9\" (UID: \"09d56c80-79bf-4e01-968f-3beab184830f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"
Oct 30 19:55:26.665664 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.665628    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wv8f\" (UniqueName: \"kubernetes.io/projected/09d56c80-79bf-4e01-968f-3beab184830f-kube-api-access-6wv8f\") pod \"maas-api-revoke-sub-f40b9632-9nhn9\" (UID: \"09d56c80-79bf-4e01-968f-3beab184830f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"
Oct 30 19:55:26.666550 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.622301    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:26.677884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.638605    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d56c80-79bf-4e01-968f-3beab184830f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f40b9632-9nhn9\" (UID: \"09d56c80-79bf-4e01-968f-3beab184830f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"
Oct 30 19:55:26.677884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.622540    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:26.677884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.675302    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": context deadline exceeded"
Oct 30 19:55:26.740074 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.689740    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:26.768808 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.768757    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d56c80-79bf-4e01-968f-3beab184830f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f40b9632-9nhn9\" (UID: \"09d56c80-79bf-4e01-968f-3beab184830f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"
Oct 30 19:55:26.770355 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.770325    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:55:26.770571 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.770549    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:55:26.799883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.791732    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:55:26.828316 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.798325    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wv8f\" (UniqueName: \"kubernetes.io/projected/09d56c80-79bf-4e01-968f-3beab184830f-kube-api-access-6wv8f\") pod \"maas-api-revoke-sub-f40b9632-9nhn9\" (UID: \"09d56c80-79bf-4e01-968f-3beab184830f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"
Oct 30 19:55:26.863752 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.863660    4823 generic.go:356] "Generic (PLEG): container finished" podID="2f10e1fd-8a47-4d0a-8eb0-85d6763c7515" containerID="86139569599fb137c7e8e6a9225968db289ddc573f9de8674a5e34c98db59a4b" exitCode=0
Oct 30 19:55:26.865125 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.865093    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867" event={"ID":"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515","Type":"ContainerDied","Data":"86139569599fb137c7e8e6a9225968db289ddc573f9de8674a5e34c98db59a4b"}
Oct 30 19:55:26.907552 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.907519    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:55:26.907827 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.907808    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:55:26.939515 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.939474    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:26.977377 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.977343    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:55:26.982381 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.982346    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:55:26.990111 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.990084    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:55:26.991339 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:26.990346    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:27.000906 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:27.000853    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"
Oct 30 19:55:27.033518 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:27.033489    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:55:27.035319 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:27.035290    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:55:27.035552 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:27.035524    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:55:27.244514 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:27.243999    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:55:27.286300 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:27.283269    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:27.335189 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:27.333762    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:55:27.372938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:27.363627    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:27.682289 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:27.682013    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:55:27.682289 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:27.682067    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:55:27.684442 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:27.683639    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:55:28.132483 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.132429    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867" event={"ID":"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515","Type":"ContainerDied","Data":"7c5ab0c7bde578bb8804e0ec70f3cb7f6e7955f2f3554fc82c0f57289c4469d5"}
Oct 30 19:55:28.132770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.132742    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c5ab0c7bde578bb8804e0ec70f3cb7f6e7955f2f3554fc82c0f57289c4469d5"
Oct 30 19:55:28.133039 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.133019    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"
Oct 30 19:55:28.410894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.409299    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-kube-api-access-jvssk\" (UniqueName: \"kubernetes.io/projected/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-kube-api-access-jvssk\") pod \"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515\" (UID: \"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515\") "
Oct 30 19:55:28.410894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.409427    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-maas-api-openshift-ca\") pod \"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515\" (UID: \"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515\") "
Oct 30 19:55:28.410894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.409513    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-maas-api-xks-ca\") pod \"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515\" (UID: \"2f10e1fd-8a47-4d0a-8eb0-85d6763c7515\") "
Oct 30 19:55:28.479133 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.479090    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-maas-api-openshift-ca" pod "2f10e1fd-8a47-4d0a-8eb0-85d6763c7515" (UID: "2f10e1fd-8a47-4d0a-8eb0-85d6763c7515"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:28.556395 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.555433    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.570262 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.568335    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:28.570262 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.570078    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-kube-api-access-jvssk" pod "2f10e1fd-8a47-4d0a-8eb0-85d6763c7515" (UID: "2f10e1fd-8a47-4d0a-8eb0-85d6763c7515"). InnerVolumeSpecName "kube-api-access-jvssk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:28.570262 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.570213    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-maas-api-xks-ca" pod "2f10e1fd-8a47-4d0a-8eb0-85d6763c7515" (UID: "2f10e1fd-8a47-4d0a-8eb0-85d6763c7515"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:28.572129 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.572070    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 19:55:28.572845 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.572826    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:28.575623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.575600    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 19:55:28.649514 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.649478    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"]
Oct 30 19:55:28.679030 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.661782    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.679272 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.679248    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvssk\" (UniqueName: \"kubernetes.io/projected/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515-kube-api-access-jvssk\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:28.681408 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.681373    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:55:28.681565 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.681457    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:55:28.681840 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.681813    4823 cpu_manager.go:414] "containerMap: removing container" podUID="2f10e1fd-8a47-4d0a-8eb0-85d6763c7515" containerName="revoke-keys"
Oct 30 19:55:28.682140 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.682120    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f10e1fd-8a47-4d0a-8eb0-85d6763c7515" containerName="revoke-keys"
Oct 30 19:55:28.682740 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.682722    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f10e1fd-8a47-4d0a-8eb0-85d6763c7515" containerName="revoke-keys"
Oct 30 19:55:28.693830 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.693799    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"]
Oct 30 19:55:28.694187 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.694148    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"
Oct 30 19:55:28.732581 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.732549    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"]
Oct 30 19:55:28.750786 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:55:28.750744    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d56c80_79bf_4e01_968f_3beab184830f.slice/crio-73159c302b105c2cd8ffff9566ccb3a980192b82ea58191fd1faf20f82c731b1 WatchSource:0}: Error finding container 73159c302b105c2cd8ffff9566ccb3a980192b82ea58191fd1faf20f82c731b1: Status 404 returned error can't find the container with id 73159c302b105c2cd8ffff9566ccb3a980192b82ea58191fd1faf20f82c731b1
Oct 30 19:55:28.781430 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.781380    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb581435-f2skc\" (UID: \"a2ba9b3b-5381-47f8-bf01-b4f9fa104713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"
Oct 30 19:55:28.781598 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.781577    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb581435-f2skc\" (UID: \"a2ba9b3b-5381-47f8-bf01-b4f9fa104713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"
Oct 30 19:55:28.781677 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.781640    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79kwl\" (UniqueName: \"kubernetes.io/projected/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-kube-api-access-79kwl\") pod \"maas-api-revoke-sub-fb581435-f2skc\" (UID: \"a2ba9b3b-5381-47f8-bf01-b4f9fa104713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"
Oct 30 19:55:28.888128 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.888092    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"]
Oct 30 19:55:28.894024 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.893991    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb581435-f2skc\" (UID: \"a2ba9b3b-5381-47f8-bf01-b4f9fa104713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"
Oct 30 19:55:28.894162 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.894056    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79kwl\" (UniqueName: \"kubernetes.io/projected/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-kube-api-access-79kwl\") pod \"maas-api-revoke-sub-fb581435-f2skc\" (UID: \"a2ba9b3b-5381-47f8-bf01-b4f9fa104713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"
Oct 30 19:55:28.894346 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.894326    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb581435-f2skc\" (UID: \"a2ba9b3b-5381-47f8-bf01-b4f9fa104713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"
Oct 30 19:55:28.896631 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.894796    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"
Oct 30 19:55:28.897318 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.895493    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb581435-f2skc\" (UID: \"a2ba9b3b-5381-47f8-bf01-b4f9fa104713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"
Oct 30 19:55:28.910088 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.910023    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb581435-f2skc\" (UID: \"a2ba9b3b-5381-47f8-bf01-b4f9fa104713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"
Oct 30 19:55:28.932403 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.932295    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79kwl\" (UniqueName: \"kubernetes.io/projected/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-kube-api-access-79kwl\") pod \"maas-api-revoke-sub-fb581435-f2skc\" (UID: \"a2ba9b3b-5381-47f8-bf01-b4f9fa104713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"
Oct 30 19:55:28.949145 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.949053    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"]
Oct 30 19:55:28.996477 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.996432    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5spm\" (UniqueName: \"kubernetes.io/projected/614afb60-e329-43eb-a03b-f23c2456580b-kube-api-access-f5spm\") pod \"maas-api-revoke-keys-e2e-ait-df345a65-sscqh\" (UID: \"614afb60-e329-43eb-a03b-f23c2456580b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"
Oct 30 19:55:28.996654 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.996520    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614afb60-e329-43eb-a03b-f23c2456580b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-df345a65-sscqh\" (UID: \"614afb60-e329-43eb-a03b-f23c2456580b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"
Oct 30 19:55:28.996654 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:28.996575    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614afb60-e329-43eb-a03b-f23c2456580b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-df345a65-sscqh\" (UID: \"614afb60-e329-43eb-a03b-f23c2456580b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"
Oct 30 19:55:29.097777 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:29.097736    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"
Oct 30 19:55:29.098383 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:29.098354    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5spm\" (UniqueName: \"kubernetes.io/projected/614afb60-e329-43eb-a03b-f23c2456580b-kube-api-access-f5spm\") pod \"maas-api-revoke-keys-e2e-ait-df345a65-sscqh\" (UID: \"614afb60-e329-43eb-a03b-f23c2456580b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"
Oct 30 19:55:29.098518 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:29.098475    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614afb60-e329-43eb-a03b-f23c2456580b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-df345a65-sscqh\" (UID: \"614afb60-e329-43eb-a03b-f23c2456580b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"
Oct 30 19:55:29.098594 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:29.098554    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614afb60-e329-43eb-a03b-f23c2456580b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-df345a65-sscqh\" (UID: \"614afb60-e329-43eb-a03b-f23c2456580b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"
Oct 30 19:55:29.099911 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:29.099883    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614afb60-e329-43eb-a03b-f23c2456580b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-df345a65-sscqh\" (UID: \"614afb60-e329-43eb-a03b-f23c2456580b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"
Oct 30 19:55:29.102050 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:29.102016    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614afb60-e329-43eb-a03b-f23c2456580b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-df345a65-sscqh\" (UID: \"614afb60-e329-43eb-a03b-f23c2456580b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"
Oct 30 19:55:29.111295 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:29.111264    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5spm\" (UniqueName: \"kubernetes.io/projected/614afb60-e329-43eb-a03b-f23c2456580b-kube-api-access-f5spm\") pod \"maas-api-revoke-keys-e2e-ait-df345a65-sscqh\" (UID: \"614afb60-e329-43eb-a03b-f23c2456580b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"
Oct 30 19:55:29.149116 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:29.149065    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9" event={"ID":"09d56c80-79bf-4e01-968f-3beab184830f","Type":"ContainerStarted","Data":"73159c302b105c2cd8ffff9566ccb3a980192b82ea58191fd1faf20f82c731b1"}
Oct 30 19:55:29.149300 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:29.149206    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"
Oct 30 19:55:29.283457 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:29.283182    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"
Oct 30 19:55:29.949699 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:29.946893    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"]
Oct 30 19:55:30.224055 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:30.214342    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9" event={"ID":"09d56c80-79bf-4e01-968f-3beab184830f","Type":"ContainerStarted","Data":"bd93abdec51aedfe2c151cbf3f15fc1f85c51eb34e37c357d751279f080e7da4"}
Oct 30 19:55:30.224055 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:30.221099    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc" event={"ID":"a2ba9b3b-5381-47f8-bf01-b4f9fa104713","Type":"ContainerStarted","Data":"89ad900a8758c50508dfde19b8cba12e5a02645ae040dabcb5ac8bddbc937d97"}
Oct 30 19:55:30.331409 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:30.330461    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9" podStartSLOduration=5.330439817 podStartE2EDuration="5.330439817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:30.26860771 +0000 UTC m=+1530.161222082" watchObservedRunningTime="2026-10-30 19:55:30.330439817 +0000 UTC m=+1530.223054143"
Oct 30 19:55:30.333584 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:30.333537    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"]
Oct 30 19:55:30.729041 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:30.729006    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:30.729184 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:30.729082    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:31.236784 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:31.236683    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh" event={"ID":"614afb60-e329-43eb-a03b-f23c2456580b","Type":"ContainerStarted","Data":"ca570e1486cbc1b6c8b4201a51f41b9111346a63040830e0fd4a2f9709408569"}
Oct 30 19:55:31.237446 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:31.237420    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh" event={"ID":"614afb60-e329-43eb-a03b-f23c2456580b","Type":"ContainerStarted","Data":"4586f6a8359b7c90c564d0c61d64371695b6b9a5b13de7f71a1801a752aca2a2"}
Oct 30 19:55:31.239570 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:31.239542    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc" event={"ID":"a2ba9b3b-5381-47f8-bf01-b4f9fa104713","Type":"ContainerStarted","Data":"d2eb316fb78dd12d3a78b03a3a51e8633c8f3cb5ad9f1417fd59ca126eb56049"}
Oct 30 19:55:31.241517 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:31.241496    4823 generic.go:356] "Generic (PLEG): container finished" podID="09d56c80-79bf-4e01-968f-3beab184830f" containerID="bd93abdec51aedfe2c151cbf3f15fc1f85c51eb34e37c357d751279f080e7da4" exitCode=0
Oct 30 19:55:31.241785 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:31.241628    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9" event={"ID":"09d56c80-79bf-4e01-968f-3beab184830f","Type":"ContainerDied","Data":"bd93abdec51aedfe2c151cbf3f15fc1f85c51eb34e37c357d751279f080e7da4"}
Oct 30 19:55:31.292244 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:31.292178    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh" podStartSLOduration=3.292158593 podStartE2EDuration="3.292158593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:31.290620058 +0000 UTC m=+1531.183234382" watchObservedRunningTime="2026-10-30 19:55:31.292158593 +0000 UTC m=+1531.184772918"
Oct 30 19:55:31.449771 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:31.449718    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc" podStartSLOduration=3.449703568 podStartE2EDuration="3.449703568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:31.448764593 +0000 UTC m=+1531.341378917" watchObservedRunningTime="2026-10-30 19:55:31.449703568 +0000 UTC m=+1531.342317928"
Oct 30 19:55:31.461220 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:31.461187    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:31.465951 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:31.465920    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 19:55:32.019439 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:55:32.019359    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:55:32.254184 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:32.254146    4823 generic.go:356] "Generic (PLEG): container finished" podID="614afb60-e329-43eb-a03b-f23c2456580b" containerID="ca570e1486cbc1b6c8b4201a51f41b9111346a63040830e0fd4a2f9709408569" exitCode=0
Oct 30 19:55:32.254597 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:32.254242    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh" event={"ID":"614afb60-e329-43eb-a03b-f23c2456580b","Type":"ContainerDied","Data":"ca570e1486cbc1b6c8b4201a51f41b9111346a63040830e0fd4a2f9709408569"}
Oct 30 19:55:32.256460 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:32.256435    4823 generic.go:356] "Generic (PLEG): container finished" podID="a2ba9b3b-5381-47f8-bf01-b4f9fa104713" containerID="d2eb316fb78dd12d3a78b03a3a51e8633c8f3cb5ad9f1417fd59ca126eb56049" exitCode=0
Oct 30 19:55:32.256571 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:32.256513    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc" event={"ID":"a2ba9b3b-5381-47f8-bf01-b4f9fa104713","Type":"ContainerDied","Data":"d2eb316fb78dd12d3a78b03a3a51e8633c8f3cb5ad9f1417fd59ca126eb56049"}
Oct 30 19:55:32.505140 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:32.505111    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"
Oct 30 19:55:32.618217 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:32.618178    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09d56c80-79bf-4e01-968f-3beab184830f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d56c80-79bf-4e01-968f-3beab184830f-maas-api-xks-ca\") pod \"09d56c80-79bf-4e01-968f-3beab184830f\" (UID: \"09d56c80-79bf-4e01-968f-3beab184830f\") "
Oct 30 19:55:32.618413 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:32.618340    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d56c80-79bf-4e01-968f-3beab184830f-kube-api-access-6wv8f\" (UniqueName: \"kubernetes.io/projected/09d56c80-79bf-4e01-968f-3beab184830f-kube-api-access-6wv8f\") pod \"09d56c80-79bf-4e01-968f-3beab184830f\" (UID: \"09d56c80-79bf-4e01-968f-3beab184830f\") "
Oct 30 19:55:32.618644 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:32.618622    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d56c80-79bf-4e01-968f-3beab184830f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d56c80-79bf-4e01-968f-3beab184830f-maas-api-openshift-ca\") pod \"09d56c80-79bf-4e01-968f-3beab184830f\" (UID: \"09d56c80-79bf-4e01-968f-3beab184830f\") "
Oct 30 19:55:32.619089 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:32.619055    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d56c80-79bf-4e01-968f-3beab184830f-maas-api-openshift-ca" pod "09d56c80-79bf-4e01-968f-3beab184830f" (UID: "09d56c80-79bf-4e01-968f-3beab184830f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:32.619434 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:32.619410    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d56c80-79bf-4e01-968f-3beab184830f-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:32.620603 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:32.620572    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d56c80-79bf-4e01-968f-3beab184830f-maas-api-xks-ca" pod "09d56c80-79bf-4e01-968f-3beab184830f" (UID: "09d56c80-79bf-4e01-968f-3beab184830f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:32.621245 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:32.621221    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d56c80-79bf-4e01-968f-3beab184830f-kube-api-access-6wv8f" pod "09d56c80-79bf-4e01-968f-3beab184830f" (UID: "09d56c80-79bf-4e01-968f-3beab184830f"). InnerVolumeSpecName "kube-api-access-6wv8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:32.738497 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:32.738456    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wv8f\" (UniqueName: \"kubernetes.io/projected/09d56c80-79bf-4e01-968f-3beab184830f-kube-api-access-6wv8f\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:32.738720 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:32.738703    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d56c80-79bf-4e01-968f-3beab184830f-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:32.858020 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:32.857983    4823 patch_prober.go:29] interesting pod/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:32.858194 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:32.858051    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n" podUID="0947f714-d384-4e21-829a-cb2ab5f70636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:33.272335 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.272302    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"
Oct 30 19:55:33.273259 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.272308    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9" event={"ID":"09d56c80-79bf-4e01-968f-3beab184830f","Type":"ContainerDied","Data":"73159c302b105c2cd8ffff9566ccb3a980192b82ea58191fd1faf20f82c731b1"}
Oct 30 19:55:33.273259 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.272446    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73159c302b105c2cd8ffff9566ccb3a980192b82ea58191fd1faf20f82c731b1"
Oct 30 19:55:33.502760 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.502722    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"
Oct 30 19:55:33.535129 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.535067    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"
Oct 30 19:55:33.653921 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.653854    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-kube-api-access-79kwl\" (UniqueName: \"kubernetes.io/projected/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-kube-api-access-79kwl\") pod \"a2ba9b3b-5381-47f8-bf01-b4f9fa104713\" (UID: \"a2ba9b3b-5381-47f8-bf01-b4f9fa104713\") "
Oct 30 19:55:33.654105 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.654046    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/614afb60-e329-43eb-a03b-f23c2456580b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614afb60-e329-43eb-a03b-f23c2456580b-maas-api-openshift-ca\") pod \"614afb60-e329-43eb-a03b-f23c2456580b\" (UID: \"614afb60-e329-43eb-a03b-f23c2456580b\") "
Oct 30 19:55:33.654176 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.654136    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-maas-api-openshift-ca\") pod \"a2ba9b3b-5381-47f8-bf01-b4f9fa104713\" (UID: \"a2ba9b3b-5381-47f8-bf01-b4f9fa104713\") "
Oct 30 19:55:33.654240 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.654229    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-maas-api-xks-ca\") pod \"a2ba9b3b-5381-47f8-bf01-b4f9fa104713\" (UID: \"a2ba9b3b-5381-47f8-bf01-b4f9fa104713\") "
Oct 30 19:55:33.654422 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.654314    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/614afb60-e329-43eb-a03b-f23c2456580b-kube-api-access-f5spm\" (UniqueName: \"kubernetes.io/projected/614afb60-e329-43eb-a03b-f23c2456580b-kube-api-access-f5spm\") pod \"614afb60-e329-43eb-a03b-f23c2456580b\" (UID: \"614afb60-e329-43eb-a03b-f23c2456580b\") "
Oct 30 19:55:33.654482 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.654461    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/614afb60-e329-43eb-a03b-f23c2456580b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614afb60-e329-43eb-a03b-f23c2456580b-maas-api-xks-ca\") pod \"614afb60-e329-43eb-a03b-f23c2456580b\" (UID: \"614afb60-e329-43eb-a03b-f23c2456580b\") "
Oct 30 19:55:33.654532 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.654502    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/614afb60-e329-43eb-a03b-f23c2456580b-maas-api-openshift-ca" pod "614afb60-e329-43eb-a03b-f23c2456580b" (UID: "614afb60-e329-43eb-a03b-f23c2456580b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:33.654672 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.654639    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-maas-api-openshift-ca" pod "a2ba9b3b-5381-47f8-bf01-b4f9fa104713" (UID: "a2ba9b3b-5381-47f8-bf01-b4f9fa104713"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:33.655395 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.655365    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614afb60-e329-43eb-a03b-f23c2456580b-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:33.655502 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.655401    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:33.656551 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.656524    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614afb60-e329-43eb-a03b-f23c2456580b-kube-api-access-f5spm" pod "614afb60-e329-43eb-a03b-f23c2456580b" (UID: "614afb60-e329-43eb-a03b-f23c2456580b"). InnerVolumeSpecName "kube-api-access-f5spm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:33.656991 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.656966    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/614afb60-e329-43eb-a03b-f23c2456580b-maas-api-xks-ca" pod "614afb60-e329-43eb-a03b-f23c2456580b" (UID: "614afb60-e329-43eb-a03b-f23c2456580b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:33.657589 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.657562    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-kube-api-access-79kwl" pod "a2ba9b3b-5381-47f8-bf01-b4f9fa104713" (UID: "a2ba9b3b-5381-47f8-bf01-b4f9fa104713"). InnerVolumeSpecName "kube-api-access-79kwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:33.657775 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.657753    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-maas-api-xks-ca" pod "a2ba9b3b-5381-47f8-bf01-b4f9fa104713" (UID: "a2ba9b3b-5381-47f8-bf01-b4f9fa104713"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:33.757172 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.757136    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614afb60-e329-43eb-a03b-f23c2456580b-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:33.757172 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.757166    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79kwl\" (UniqueName: \"kubernetes.io/projected/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-kube-api-access-79kwl\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:33.757172 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.757180    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ba9b3b-5381-47f8-bf01-b4f9fa104713-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:33.757389 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:33.757189    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5spm\" (UniqueName: \"kubernetes.io/projected/614afb60-e329-43eb-a03b-f23c2456580b-kube-api-access-f5spm\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:34.378798 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:34.378764    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc" event={"ID":"a2ba9b3b-5381-47f8-bf01-b4f9fa104713","Type":"ContainerDied","Data":"89ad900a8758c50508dfde19b8cba12e5a02645ae040dabcb5ac8bddbc937d97"}
Oct 30 19:55:34.379995 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:34.379970    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89ad900a8758c50508dfde19b8cba12e5a02645ae040dabcb5ac8bddbc937d97"
Oct 30 19:55:34.379995 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:34.379282    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"
Oct 30 19:55:34.386456 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:34.386438    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"
Oct 30 19:55:34.386645 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:34.386457    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh" event={"ID":"614afb60-e329-43eb-a03b-f23c2456580b","Type":"ContainerDied","Data":"4586f6a8359b7c90c564d0c61d64371695b6b9a5b13de7f71a1801a752aca2a2"}
Oct 30 19:55:34.386699 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:34.386667    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4586f6a8359b7c90c564d0c61d64371695b6b9a5b13de7f71a1801a752aca2a2"
Oct 30 19:55:35.397031 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.396997    4823 generic.go:356] "Generic (PLEG): container finished" podID="0947f714-d384-4e21-829a-cb2ab5f70636" containerID="8d49214b8251430b7e39f9e4d35fb7cb5b3683d8741605019815cf20fc5940e0" exitCode=0
Oct 30 19:55:35.397484 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.397069    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n" event={"ID":"0947f714-d384-4e21-829a-cb2ab5f70636","Type":"ContainerDied","Data":"8d49214b8251430b7e39f9e4d35fb7cb5b3683d8741605019815cf20fc5940e0"}
Oct 30 19:55:35.397545 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.397529    4823 scope.go:122] "RemoveContainer" containerID="8d49214b8251430b7e39f9e4d35fb7cb5b3683d8741605019815cf20fc5940e0"
Oct 30 19:55:35.399291 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.399264    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:55:35.399360 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.399333    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:55:35.399436 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.399365    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:55:35.546890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.546814    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"]
Oct 30 19:55:35.572996 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.557199    4823 cpu_manager.go:414] "containerMap: removing container" podUID="a2ba9b3b-5381-47f8-bf01-b4f9fa104713" containerName="revoke-keys"
Oct 30 19:55:35.572996 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.557236    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ba9b3b-5381-47f8-bf01-b4f9fa104713" containerName="revoke-keys"
Oct 30 19:55:35.572996 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.557278    4823 cpu_manager.go:414] "containerMap: removing container" podUID="614afb60-e329-43eb-a03b-f23c2456580b" containerName="revoke-keys"
Oct 30 19:55:35.572996 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.557287    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614afb60-e329-43eb-a03b-f23c2456580b" containerName="revoke-keys"
Oct 30 19:55:35.572996 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.557331    4823 cpu_manager.go:414] "containerMap: removing container" podUID="09d56c80-79bf-4e01-968f-3beab184830f" containerName="revoke-keys"
Oct 30 19:55:35.572996 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.557339    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d56c80-79bf-4e01-968f-3beab184830f" containerName="revoke-keys"
Oct 30 19:55:35.572996 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.557538    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ba9b3b-5381-47f8-bf01-b4f9fa104713" containerName="revoke-keys"
Oct 30 19:55:35.572996 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.557552    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="614afb60-e329-43eb-a03b-f23c2456580b" containerName="revoke-keys"
Oct 30 19:55:35.572996 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.557567    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d56c80-79bf-4e01-968f-3beab184830f" containerName="revoke-keys"
Oct 30 19:55:35.586926 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.585430    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"]
Oct 30 19:55:35.586926 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.585561    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"
Oct 30 19:55:35.588784 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.588762    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:55:35.588947 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.588916    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:55:35.589095 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.589080    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:55:35.589160 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.589147    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:55:35.612072 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.610431    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805a1f72-lmfnd\" (UID: \"c5aabcbc-69dc-4da8-8589-5305dd2f77d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"
Oct 30 19:55:35.612072 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.610637    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805a1f72-lmfnd\" (UID: \"c5aabcbc-69dc-4da8-8589-5305dd2f77d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"
Oct 30 19:55:35.612072 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.610755    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlqfq\" (UniqueName: \"kubernetes.io/projected/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-kube-api-access-hlqfq\") pod \"maas-api-revoke-sub-805a1f72-lmfnd\" (UID: \"c5aabcbc-69dc-4da8-8589-5305dd2f77d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"
Oct 30 19:55:35.712425 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.712342    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805a1f72-lmfnd\" (UID: \"c5aabcbc-69dc-4da8-8589-5305dd2f77d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"
Oct 30 19:55:35.712583 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.712496    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805a1f72-lmfnd\" (UID: \"c5aabcbc-69dc-4da8-8589-5305dd2f77d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"
Oct 30 19:55:35.712651 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.712598    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlqfq\" (UniqueName: \"kubernetes.io/projected/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-kube-api-access-hlqfq\") pod \"maas-api-revoke-sub-805a1f72-lmfnd\" (UID: \"c5aabcbc-69dc-4da8-8589-5305dd2f77d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"
Oct 30 19:55:35.713094 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.713066    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805a1f72-lmfnd\" (UID: \"c5aabcbc-69dc-4da8-8589-5305dd2f77d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"
Oct 30 19:55:35.715034 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.715008    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805a1f72-lmfnd\" (UID: \"c5aabcbc-69dc-4da8-8589-5305dd2f77d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"
Oct 30 19:55:35.726466 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.726433    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlqfq\" (UniqueName: \"kubernetes.io/projected/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-kube-api-access-hlqfq\") pod \"maas-api-revoke-sub-805a1f72-lmfnd\" (UID: \"c5aabcbc-69dc-4da8-8589-5305dd2f77d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"
Oct 30 19:55:35.919039 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:35.919010    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"
Oct 30 19:55:36.215838 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:55:36.215805    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5aabcbc_69dc_4da8_8589_5305dd2f77d0.slice/crio-f95de1831c9185b4fa539ce8672ef9f668c8482607b473b82cf4fe39989dbab1 WatchSource:0}: Error finding container f95de1831c9185b4fa539ce8672ef9f668c8482607b473b82cf4fe39989dbab1: Status 404 returned error can't find the container with id f95de1831c9185b4fa539ce8672ef9f668c8482607b473b82cf4fe39989dbab1
Oct 30 19:55:36.219524 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:36.219499    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"]
Oct 30 19:55:36.407584 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:36.407546    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd" event={"ID":"c5aabcbc-69dc-4da8-8589-5305dd2f77d0","Type":"ContainerStarted","Data":"f95de1831c9185b4fa539ce8672ef9f668c8482607b473b82cf4fe39989dbab1"}
Oct 30 19:55:36.410173 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:36.410137    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n" event={"ID":"0947f714-d384-4e21-829a-cb2ab5f70636","Type":"ContainerStarted","Data":"365876976e2dfe8654db03f62ce198063b7f3908ad4d3872739fa3fd2d61e9a9"}
Oct 30 19:55:36.689210 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:36.689170    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:55:36.689428 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:36.689398    4823 patch_prober.go:29] interesting pod/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:55:36.689534 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:36.689449    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n" podUID="0947f714-d384-4e21-829a-cb2ab5f70636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 19:55:37.384062 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.384025    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"]
Oct 30 19:55:37.388716 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.388694    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"
Oct 30 19:55:37.390695 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.390668    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"]
Oct 30 19:55:37.423732 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.423699    4823 generic.go:356] "Generic (PLEG): container finished" podID="c5aabcbc-69dc-4da8-8589-5305dd2f77d0" containerID="057cb99d26bd8d39bf792ed6c7300c61432f51b9f682858c16cf2152d9460684" exitCode=0
Oct 30 19:55:37.424273 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.423784    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd" event={"ID":"c5aabcbc-69dc-4da8-8589-5305dd2f77d0","Type":"ContainerDied","Data":"057cb99d26bd8d39bf792ed6c7300c61432f51b9f682858c16cf2152d9460684"}
Oct 30 19:55:37.485069 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.485025    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208b36f6-c28f-4e09-a03f-19cffd9a695f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea210528-vkwm5\" (UID: \"208b36f6-c28f-4e09-a03f-19cffd9a695f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"
Oct 30 19:55:37.485299 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.485275    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208b36f6-c28f-4e09-a03f-19cffd9a695f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea210528-vkwm5\" (UID: \"208b36f6-c28f-4e09-a03f-19cffd9a695f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"
Oct 30 19:55:37.485742 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.485719    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hvfj\" (UniqueName: \"kubernetes.io/projected/208b36f6-c28f-4e09-a03f-19cffd9a695f-kube-api-access-9hvfj\") pod \"maas-api-revoke-keys-e2e-ait-ea210528-vkwm5\" (UID: \"208b36f6-c28f-4e09-a03f-19cffd9a695f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"
Oct 30 19:55:37.587651 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.587565    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hvfj\" (UniqueName: \"kubernetes.io/projected/208b36f6-c28f-4e09-a03f-19cffd9a695f-kube-api-access-9hvfj\") pod \"maas-api-revoke-keys-e2e-ait-ea210528-vkwm5\" (UID: \"208b36f6-c28f-4e09-a03f-19cffd9a695f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"
Oct 30 19:55:37.587651 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.587644    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208b36f6-c28f-4e09-a03f-19cffd9a695f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea210528-vkwm5\" (UID: \"208b36f6-c28f-4e09-a03f-19cffd9a695f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"
Oct 30 19:55:37.587958 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.587741    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208b36f6-c28f-4e09-a03f-19cffd9a695f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea210528-vkwm5\" (UID: \"208b36f6-c28f-4e09-a03f-19cffd9a695f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"
Oct 30 19:55:37.588492 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.588464    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208b36f6-c28f-4e09-a03f-19cffd9a695f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea210528-vkwm5\" (UID: \"208b36f6-c28f-4e09-a03f-19cffd9a695f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"
Oct 30 19:55:37.590092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.590070    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208b36f6-c28f-4e09-a03f-19cffd9a695f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea210528-vkwm5\" (UID: \"208b36f6-c28f-4e09-a03f-19cffd9a695f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"
Oct 30 19:55:37.599289 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.599268    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hvfj\" (UniqueName: \"kubernetes.io/projected/208b36f6-c28f-4e09-a03f-19cffd9a695f-kube-api-access-9hvfj\") pod \"maas-api-revoke-keys-e2e-ait-ea210528-vkwm5\" (UID: \"208b36f6-c28f-4e09-a03f-19cffd9a695f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"
Oct 30 19:55:37.689600 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.689561    4823 patch_prober.go:29] interesting pod/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:55:37.689787 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.689626    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n" podUID="0947f714-d384-4e21-829a-cb2ab5f70636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 19:55:37.704751 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:37.704725    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"
Oct 30 19:55:38.437695 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:38.437663    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"]
Oct 30 19:55:38.690401 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:38.690368    4823 patch_prober.go:29] interesting pod/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:55:38.690536 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:38.690435    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n" podUID="0947f714-d384-4e21-829a-cb2ab5f70636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 19:55:38.705984 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:38.705959    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"
Oct 30 19:55:38.804906 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:38.804858    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-kube-api-access-hlqfq\" (UniqueName: \"kubernetes.io/projected/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-kube-api-access-hlqfq\") pod \"c5aabcbc-69dc-4da8-8589-5305dd2f77d0\" (UID: \"c5aabcbc-69dc-4da8-8589-5305dd2f77d0\") "
Oct 30 19:55:38.805107 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:38.804920    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-maas-api-xks-ca\") pod \"c5aabcbc-69dc-4da8-8589-5305dd2f77d0\" (UID: \"c5aabcbc-69dc-4da8-8589-5305dd2f77d0\") "
Oct 30 19:55:38.805303 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:38.805284    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-maas-api-openshift-ca\") pod \"c5aabcbc-69dc-4da8-8589-5305dd2f77d0\" (UID: \"c5aabcbc-69dc-4da8-8589-5305dd2f77d0\") "
Oct 30 19:55:38.805684 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:38.805658    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-maas-api-openshift-ca" pod "c5aabcbc-69dc-4da8-8589-5305dd2f77d0" (UID: "c5aabcbc-69dc-4da8-8589-5305dd2f77d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:38.805968 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:38.805950    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:38.807123 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:38.807098    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-kube-api-access-hlqfq" pod "c5aabcbc-69dc-4da8-8589-5305dd2f77d0" (UID: "c5aabcbc-69dc-4da8-8589-5305dd2f77d0"). InnerVolumeSpecName "kube-api-access-hlqfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:38.807358 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:38.807333    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-maas-api-xks-ca" pod "c5aabcbc-69dc-4da8-8589-5305dd2f77d0" (UID: "c5aabcbc-69dc-4da8-8589-5305dd2f77d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:38.907539 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:38.907503    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlqfq\" (UniqueName: \"kubernetes.io/projected/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-kube-api-access-hlqfq\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:38.907539 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:38.907535    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5aabcbc-69dc-4da8-8589-5305dd2f77d0-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:39.481819 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:39.480133    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd" event={"ID":"c5aabcbc-69dc-4da8-8589-5305dd2f77d0","Type":"ContainerDied","Data":"f95de1831c9185b4fa539ce8672ef9f668c8482607b473b82cf4fe39989dbab1"}
Oct 30 19:55:39.481819 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:39.480227    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f95de1831c9185b4fa539ce8672ef9f668c8482607b473b82cf4fe39989dbab1"
Oct 30 19:55:39.481819 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:39.480746    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"
Oct 30 19:55:39.485199 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:39.485153    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5" event={"ID":"208b36f6-c28f-4e09-a03f-19cffd9a695f","Type":"ContainerStarted","Data":"7347190d7c58d04089d4081a518ec6a76ec37dbc2031013025fff8f649fe55b6"}
Oct 30 19:55:39.485338 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:39.485227    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5" event={"ID":"208b36f6-c28f-4e09-a03f-19cffd9a695f","Type":"ContainerStarted","Data":"d7b77ded7fed9d8c1e8f30439e79edf1f8ee161eaf6a27b9d0b5c0da3396c80a"}
Oct 30 19:55:39.504365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:39.504317    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5" podStartSLOduration=2.504305261 podStartE2EDuration="2.504305261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:39.502687123 +0000 UTC m=+1539.395301447" watchObservedRunningTime="2026-10-30 19:55:39.504305261 +0000 UTC m=+1539.396919585"
Oct 30 19:55:39.696620 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:39.695566    4823 patch_prober.go:29] interesting pod/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:55:39.696620 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:39.695623    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n" podUID="0947f714-d384-4e21-829a-cb2ab5f70636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 19:55:40.185719 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:40.185684    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"]
Oct 30 19:55:40.191614 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:40.191588    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df345a65-sscqh"]
Oct 30 19:55:40.344915 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:40.341325    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="614afb60-e329-43eb-a03b-f23c2456580b" path="/var/lib/kubelet/pods/614afb60-e329-43eb-a03b-f23c2456580b/volumes"
Oct 30 19:55:40.397373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:40.397346    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"]
Oct 30 19:55:40.397739 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:40.397710    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n" podUID="0947f714-d384-4e21-829a-cb2ab5f70636" containerName="istio-proxy" containerID="cri-o://365876976e2dfe8654db03f62ce198063b7f3908ad4d3872739fa3fd2d61e9a9" gracePeriod=30
Oct 30 19:55:40.547781 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:40.547689    4823 generic.go:356] "Generic (PLEG): container finished" podID="208b36f6-c28f-4e09-a03f-19cffd9a695f" containerID="7347190d7c58d04089d4081a518ec6a76ec37dbc2031013025fff8f649fe55b6" exitCode=0
Oct 30 19:55:40.548544 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:40.548515    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5" event={"ID":"208b36f6-c28f-4e09-a03f-19cffd9a695f","Type":"ContainerDied","Data":"7347190d7c58d04089d4081a518ec6a76ec37dbc2031013025fff8f649fe55b6"}
Oct 30 19:55:41.930373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:41.930346    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"
Oct 30 19:55:41.999981 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:41.999937    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208b36f6-c28f-4e09-a03f-19cffd9a695f-kube-api-access-9hvfj\" (UniqueName: \"kubernetes.io/projected/208b36f6-c28f-4e09-a03f-19cffd9a695f-kube-api-access-9hvfj\") pod \"208b36f6-c28f-4e09-a03f-19cffd9a695f\" (UID: \"208b36f6-c28f-4e09-a03f-19cffd9a695f\") "
Oct 30 19:55:42.000186 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.000144    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208b36f6-c28f-4e09-a03f-19cffd9a695f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208b36f6-c28f-4e09-a03f-19cffd9a695f-maas-api-openshift-ca\") pod \"208b36f6-c28f-4e09-a03f-19cffd9a695f\" (UID: \"208b36f6-c28f-4e09-a03f-19cffd9a695f\") "
Oct 30 19:55:42.000411 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.000387    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/208b36f6-c28f-4e09-a03f-19cffd9a695f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208b36f6-c28f-4e09-a03f-19cffd9a695f-maas-api-xks-ca\") pod \"208b36f6-c28f-4e09-a03f-19cffd9a695f\" (UID: \"208b36f6-c28f-4e09-a03f-19cffd9a695f\") "
Oct 30 19:55:42.000577 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.000545    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208b36f6-c28f-4e09-a03f-19cffd9a695f-maas-api-openshift-ca" pod "208b36f6-c28f-4e09-a03f-19cffd9a695f" (UID: "208b36f6-c28f-4e09-a03f-19cffd9a695f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:42.001070 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.001047    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208b36f6-c28f-4e09-a03f-19cffd9a695f-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:42.002845 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.002818    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208b36f6-c28f-4e09-a03f-19cffd9a695f-maas-api-xks-ca" pod "208b36f6-c28f-4e09-a03f-19cffd9a695f" (UID: "208b36f6-c28f-4e09-a03f-19cffd9a695f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:42.002948 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.002828    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208b36f6-c28f-4e09-a03f-19cffd9a695f-kube-api-access-9hvfj" pod "208b36f6-c28f-4e09-a03f-19cffd9a695f" (UID: "208b36f6-c28f-4e09-a03f-19cffd9a695f"). InnerVolumeSpecName "kube-api-access-9hvfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:42.260279 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.260168    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hvfj\" (UniqueName: \"kubernetes.io/projected/208b36f6-c28f-4e09-a03f-19cffd9a695f-kube-api-access-9hvfj\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:42.260507 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.260487    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208b36f6-c28f-4e09-a03f-19cffd9a695f-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:42.400373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.400336    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"]
Oct 30 19:55:42.438668 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.438619    4823 cpu_manager.go:414] "containerMap: removing container" podUID="208b36f6-c28f-4e09-a03f-19cffd9a695f" containerName="revoke-keys"
Oct 30 19:55:42.439932 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.439904    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208b36f6-c28f-4e09-a03f-19cffd9a695f" containerName="revoke-keys"
Oct 30 19:55:42.440160 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.440141    4823 cpu_manager.go:414] "containerMap: removing container" podUID="c5aabcbc-69dc-4da8-8589-5305dd2f77d0" containerName="revoke-keys"
Oct 30 19:55:42.440302 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.440286    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5aabcbc-69dc-4da8-8589-5305dd2f77d0" containerName="revoke-keys"
Oct 30 19:55:42.440699 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.440664    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="208b36f6-c28f-4e09-a03f-19cffd9a695f" containerName="revoke-keys"
Oct 30 19:55:42.440817 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.440805    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5aabcbc-69dc-4da8-8589-5305dd2f77d0" containerName="revoke-keys"
Oct 30 19:55:42.449719 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.449695    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"]
Oct 30 19:55:42.450213 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.450191    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"
Oct 30 19:55:42.578415 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.572549    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"]
Oct 30 19:55:42.578415 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.573297    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwbq6\" (UniqueName: \"kubernetes.io/projected/966ec1a3-190e-4018-b40c-2055e6aefc00-kube-api-access-dwbq6\") pod \"maas-api-revoke-keys-e2e-derive-d07cb404-scnwb\" (UID: \"966ec1a3-190e-4018-b40c-2055e6aefc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"
Oct 30 19:55:42.578671 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.578642    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966ec1a3-190e-4018-b40c-2055e6aefc00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d07cb404-scnwb\" (UID: \"966ec1a3-190e-4018-b40c-2055e6aefc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"
Oct 30 19:55:42.578763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.578745    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966ec1a3-190e-4018-b40c-2055e6aefc00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d07cb404-scnwb\" (UID: \"966ec1a3-190e-4018-b40c-2055e6aefc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"
Oct 30 19:55:42.593672 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.582005    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.593672 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.582015    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"]
Oct 30 19:55:42.601880 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.600803    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b69bf89d-gw-openshift-default-dockercfg-wzvjt\""
Oct 30 19:55:42.609118 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.608187    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5" event={"ID":"208b36f6-c28f-4e09-a03f-19cffd9a695f","Type":"ContainerDied","Data":"d7b77ded7fed9d8c1e8f30439e79edf1f8ee161eaf6a27b9d0b5c0da3396c80a"}
Oct 30 19:55:42.609118 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.608234    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7b77ded7fed9d8c1e8f30439e79edf1f8ee161eaf6a27b9d0b5c0da3396c80a"
Oct 30 19:55:42.609118 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.608286    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"
Oct 30 19:55:42.680543 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.680497    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2xw6\" (UniqueName: \"kubernetes.io/projected/622c7980-cb92-4c47-ba6a-f333e5d6256e-kube-api-access-f2xw6\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.680776 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.680568    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-token\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.680776 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.680676    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-workload-socket\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.680903 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.680783    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-envoy\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.680903 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.680851    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966ec1a3-190e-4018-b40c-2055e6aefc00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d07cb404-scnwb\" (UID: \"966ec1a3-190e-4018-b40c-2055e6aefc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"
Oct 30 19:55:42.681018 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.680995    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966ec1a3-190e-4018-b40c-2055e6aefc00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d07cb404-scnwb\" (UID: \"966ec1a3-190e-4018-b40c-2055e6aefc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"
Oct 30 19:55:42.681098 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.681079    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-workload-certs\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.681257 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.681241    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-data\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.681332 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.681299    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwbq6\" (UniqueName: \"kubernetes.io/projected/966ec1a3-190e-4018-b40c-2055e6aefc00-kube-api-access-dwbq6\") pod \"maas-api-revoke-keys-e2e-derive-d07cb404-scnwb\" (UID: \"966ec1a3-190e-4018-b40c-2055e6aefc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"
Oct 30 19:55:42.681390 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.681331    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/622c7980-cb92-4c47-ba6a-f333e5d6256e-istiod-ca-cert\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.681390 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.681370    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-podinfo\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.681494 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.681413    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-credential-socket\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.681890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.681844    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966ec1a3-190e-4018-b40c-2055e6aefc00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d07cb404-scnwb\" (UID: \"966ec1a3-190e-4018-b40c-2055e6aefc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"
Oct 30 19:55:42.683636 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.683609    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966ec1a3-190e-4018-b40c-2055e6aefc00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d07cb404-scnwb\" (UID: \"966ec1a3-190e-4018-b40c-2055e6aefc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"
Oct 30 19:55:42.697427 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.697405    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwbq6\" (UniqueName: \"kubernetes.io/projected/966ec1a3-190e-4018-b40c-2055e6aefc00-kube-api-access-dwbq6\") pod \"maas-api-revoke-keys-e2e-derive-d07cb404-scnwb\" (UID: \"966ec1a3-190e-4018-b40c-2055e6aefc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"
Oct 30 19:55:42.783241 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.783201    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-workload-certs\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.783423 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.783343    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-data\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.783423 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.783376    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/622c7980-cb92-4c47-ba6a-f333e5d6256e-istiod-ca-cert\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.783423 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.783409    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-podinfo\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.783591 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.783448    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-credential-socket\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.783591 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.783571    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2xw6\" (UniqueName: \"kubernetes.io/projected/622c7980-cb92-4c47-ba6a-f333e5d6256e-kube-api-access-f2xw6\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.783702 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.783606    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-token\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.783702 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.783640    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-workload-socket\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.783802 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.783701    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-envoy\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.783802 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.783726    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-workload-certs\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.784762 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.784732    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-credential-socket\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.785035 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.785004    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-workload-socket\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.785163 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.785095    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-data\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.785244 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.785224    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/622c7980-cb92-4c47-ba6a-f333e5d6256e-istiod-ca-cert\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.786481 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.786455    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-envoy\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.786599 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.786478    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-podinfo\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.796865 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.796840    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-token\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.797425 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.797407    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2xw6\" (UniqueName: \"kubernetes.io/projected/622c7980-cb92-4c47-ba6a-f333e5d6256e-kube-api-access-f2xw6\") pod \"e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") " pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:42.886890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.885423    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"
Oct 30 19:55:42.929279 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:42.928962    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:43.232373 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:55:43.232339    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966ec1a3_190e_4018_b40c_2055e6aefc00.slice/crio-7f67e012f3b11277c116db25b5c4950f415d25f8c61a7aebac7ae7210001674c WatchSource:0}: Error finding container 7f67e012f3b11277c116db25b5c4950f415d25f8c61a7aebac7ae7210001674c: Status 404 returned error can't find the container with id 7f67e012f3b11277c116db25b5c4950f415d25f8c61a7aebac7ae7210001674c
Oct 30 19:55:43.233399 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:43.233374    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"]
Oct 30 19:55:43.241974 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:55:43.241949    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622c7980_cb92_4c47_ba6a_f333e5d6256e.slice/crio-0f550f5e8e3f57b571c22d481ed415dc44fd46245a46e317cee58513d99fc319 WatchSource:0}: Error finding container 0f550f5e8e3f57b571c22d481ed415dc44fd46245a46e317cee58513d99fc319: Status 404 returned error can't find the container with id 0f550f5e8e3f57b571c22d481ed415dc44fd46245a46e317cee58513d99fc319
Oct 30 19:55:43.243128 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:43.243106    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"]
Oct 30 19:55:43.251658 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:43.251619    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:55:43.251736 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:43.251719    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:55:43.251814 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:43.251797    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:55:43.620433 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:43.620387    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb" event={"ID":"966ec1a3-190e-4018-b40c-2055e6aefc00","Type":"ContainerStarted","Data":"7f67e012f3b11277c116db25b5c4950f415d25f8c61a7aebac7ae7210001674c"}
Oct 30 19:55:43.622393 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:43.622359    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9" event={"ID":"622c7980-cb92-4c47-ba6a-f333e5d6256e","Type":"ContainerStarted","Data":"bc999ee4a27e237ab48f4ae52ca6746dc9659fc769d227b354c062669f573e0e"}
Oct 30 19:55:43.622516 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:43.622425    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9" event={"ID":"622c7980-cb92-4c47-ba6a-f333e5d6256e","Type":"ContainerStarted","Data":"0f550f5e8e3f57b571c22d481ed415dc44fd46245a46e317cee58513d99fc319"}
Oct 30 19:55:43.657769 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:43.657704    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9" podStartSLOduration=1.657682842 podStartE2EDuration="1.657682842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:43.646994889 +0000 UTC m=+1543.539609212" watchObservedRunningTime="2026-10-30 19:55:43.657682842 +0000 UTC m=+1543.550297166"
Oct 30 19:55:43.929615 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:43.929534    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:44.667985 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:44.667945    4823 generic.go:356] "Generic (PLEG): container finished" podID="966ec1a3-190e-4018-b40c-2055e6aefc00" containerID="5b60cfc81b0191dffc2ea9b058439c3d8b91b250b42c2b8bd540159e98d96fb7" exitCode=0
Oct 30 19:55:44.668503 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:44.667997    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb" event={"ID":"966ec1a3-190e-4018-b40c-2055e6aefc00","Type":"ContainerDied","Data":"5b60cfc81b0191dffc2ea9b058439c3d8b91b250b42c2b8bd540159e98d96fb7"}
Oct 30 19:55:44.935498 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:44.935418    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:45.784272 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:45.784238    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:45.827890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:45.826616    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:55:46.234214 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:46.234185    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"
Oct 30 19:55:46.331232 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:46.331196    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/966ec1a3-190e-4018-b40c-2055e6aefc00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966ec1a3-190e-4018-b40c-2055e6aefc00-maas-api-xks-ca\") pod \"966ec1a3-190e-4018-b40c-2055e6aefc00\" (UID: \"966ec1a3-190e-4018-b40c-2055e6aefc00\") "
Oct 30 19:55:46.331584 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:46.331565    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966ec1a3-190e-4018-b40c-2055e6aefc00-kube-api-access-dwbq6\" (UniqueName: \"kubernetes.io/projected/966ec1a3-190e-4018-b40c-2055e6aefc00-kube-api-access-dwbq6\") pod \"966ec1a3-190e-4018-b40c-2055e6aefc00\" (UID: \"966ec1a3-190e-4018-b40c-2055e6aefc00\") "
Oct 30 19:55:46.331792 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:46.331778    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/966ec1a3-190e-4018-b40c-2055e6aefc00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966ec1a3-190e-4018-b40c-2055e6aefc00-maas-api-openshift-ca\") pod \"966ec1a3-190e-4018-b40c-2055e6aefc00\" (UID: \"966ec1a3-190e-4018-b40c-2055e6aefc00\") "
Oct 30 19:55:46.332341 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:46.332289    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966ec1a3-190e-4018-b40c-2055e6aefc00-maas-api-openshift-ca" pod "966ec1a3-190e-4018-b40c-2055e6aefc00" (UID: "966ec1a3-190e-4018-b40c-2055e6aefc00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:46.332881 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:46.332842    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966ec1a3-190e-4018-b40c-2055e6aefc00-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:46.334131 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:46.334108    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966ec1a3-190e-4018-b40c-2055e6aefc00-maas-api-xks-ca" pod "966ec1a3-190e-4018-b40c-2055e6aefc00" (UID: "966ec1a3-190e-4018-b40c-2055e6aefc00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:46.334401 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:46.334359    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966ec1a3-190e-4018-b40c-2055e6aefc00-kube-api-access-dwbq6" pod "966ec1a3-190e-4018-b40c-2055e6aefc00" (UID: "966ec1a3-190e-4018-b40c-2055e6aefc00"). InnerVolumeSpecName "kube-api-access-dwbq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:46.434906 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:46.434850    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966ec1a3-190e-4018-b40c-2055e6aefc00-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:46.434906 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:46.434894    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwbq6\" (UniqueName: \"kubernetes.io/projected/966ec1a3-190e-4018-b40c-2055e6aefc00-kube-api-access-dwbq6\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:46.819058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:46.818792    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:55:46.840884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:46.838398    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"
Oct 30 19:55:46.863294 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:46.862371    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb" event={"ID":"966ec1a3-190e-4018-b40c-2055e6aefc00","Type":"ContainerDied","Data":"7f67e012f3b11277c116db25b5c4950f415d25f8c61a7aebac7ae7210001674c"}
Oct 30 19:55:46.863294 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:46.862457    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f67e012f3b11277c116db25b5c4950f415d25f8c61a7aebac7ae7210001674c"
Oct 30 19:55:47.890998 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:47.890961    4823 generic.go:356] "Generic (PLEG): container finished" podID="0947f714-d384-4e21-829a-cb2ab5f70636" containerID="365876976e2dfe8654db03f62ce198063b7f3908ad4d3872739fa3fd2d61e9a9" exitCode=0
Oct 30 19:55:47.892779 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:47.892714    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n" event={"ID":"0947f714-d384-4e21-829a-cb2ab5f70636","Type":"ContainerDied","Data":"365876976e2dfe8654db03f62ce198063b7f3908ad4d3872739fa3fd2d61e9a9"}
Oct 30 19:55:47.893125 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:47.893106    4823 scope.go:122] "RemoveContainer" containerID="8d49214b8251430b7e39f9e4d35fb7cb5b3683d8741605019815cf20fc5940e0"
Oct 30 19:55:48.578922 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.571372    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:55:48.639776 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.639742    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0947f714-d384-4e21-829a-cb2ab5f70636-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0947f714-d384-4e21-829a-cb2ab5f70636-istiod-ca-cert\") pod \"0947f714-d384-4e21-829a-cb2ab5f70636\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") "
Oct 30 19:55:48.640048 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.639917    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-workload-socket\") pod \"0947f714-d384-4e21-829a-cb2ab5f70636\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") "
Oct 30 19:55:48.640048 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.639953    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-istio-envoy\") pod \"0947f714-d384-4e21-829a-cb2ab5f70636\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") "
Oct 30 19:55:48.640048 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.639986    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-credential-socket\") pod \"0947f714-d384-4e21-829a-cb2ab5f70636\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") "
Oct 30 19:55:48.640334 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.640303    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-workload-socket" pod "0947f714-d384-4e21-829a-cb2ab5f70636" (UID: "0947f714-d384-4e21-829a-cb2ab5f70636"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:48.640398 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.640333    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-credential-socket" pod "0947f714-d384-4e21-829a-cb2ab5f70636" (UID: "0947f714-d384-4e21-829a-cb2ab5f70636"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:48.640398 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.640333    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0947f714-d384-4e21-829a-cb2ab5f70636-istiod-ca-cert" pod "0947f714-d384-4e21-829a-cb2ab5f70636" (UID: "0947f714-d384-4e21-829a-cb2ab5f70636"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:48.640541 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.640523    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0947f714-d384-4e21-829a-cb2ab5f70636-kube-api-access-zgrj4\" (UniqueName: \"kubernetes.io/projected/0947f714-d384-4e21-829a-cb2ab5f70636-kube-api-access-zgrj4\") pod \"0947f714-d384-4e21-829a-cb2ab5f70636\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") "
Oct 30 19:55:48.640603 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.640571    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-istio-data\") pod \"0947f714-d384-4e21-829a-cb2ab5f70636\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") "
Oct 30 19:55:48.640657 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.640641    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0947f714-d384-4e21-829a-cb2ab5f70636-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0947f714-d384-4e21-829a-cb2ab5f70636-istio-podinfo\") pod \"0947f714-d384-4e21-829a-cb2ab5f70636\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") "
Oct 30 19:55:48.640984 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.640955    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-istio-data" pod "0947f714-d384-4e21-829a-cb2ab5f70636" (UID: "0947f714-d384-4e21-829a-cb2ab5f70636"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:48.641104 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.641086    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-workload-certs\") pod \"0947f714-d384-4e21-829a-cb2ab5f70636\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") "
Oct 30 19:55:48.641228 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.641181    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0947f714-d384-4e21-829a-cb2ab5f70636-istio-token\" (UniqueName: \"kubernetes.io/projected/0947f714-d384-4e21-829a-cb2ab5f70636-istio-token\") pod \"0947f714-d384-4e21-829a-cb2ab5f70636\" (UID: \"0947f714-d384-4e21-829a-cb2ab5f70636\") "
Oct 30 19:55:48.641364 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.641342    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-workload-certs" pod "0947f714-d384-4e21-829a-cb2ab5f70636" (UID: "0947f714-d384-4e21-829a-cb2ab5f70636"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:48.642076 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.642054    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.642193 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.642076    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.642193 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.642090    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0947f714-d384-4e21-829a-cb2ab5f70636-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.642193 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.642103    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.642193 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.642115    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.643297 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.643257    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0947f714-d384-4e21-829a-cb2ab5f70636-kube-api-access-zgrj4" pod "0947f714-d384-4e21-829a-cb2ab5f70636" (UID: "0947f714-d384-4e21-829a-cb2ab5f70636"). InnerVolumeSpecName "kube-api-access-zgrj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:48.643297 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.643275    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0947f714-d384-4e21-829a-cb2ab5f70636-istio-podinfo" pod "0947f714-d384-4e21-829a-cb2ab5f70636" (UID: "0947f714-d384-4e21-829a-cb2ab5f70636"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:55:48.644056 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.644030    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0947f714-d384-4e21-829a-cb2ab5f70636-istio-token" pod "0947f714-d384-4e21-829a-cb2ab5f70636" (UID: "0947f714-d384-4e21-829a-cb2ab5f70636"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:48.644056 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.644031    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-istio-envoy" pod "0947f714-d384-4e21-829a-cb2ab5f70636" (UID: "0947f714-d384-4e21-829a-cb2ab5f70636"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:48.743640 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.743610    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgrj4\" (UniqueName: \"kubernetes.io/projected/0947f714-d384-4e21-829a-cb2ab5f70636-kube-api-access-zgrj4\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.743640 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.743638    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0947f714-d384-4e21-829a-cb2ab5f70636-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.743846 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.743649    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0947f714-d384-4e21-829a-cb2ab5f70636-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.743846 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.743659    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0947f714-d384-4e21-829a-cb2ab5f70636-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.910659 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.904777    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n" event={"ID":"0947f714-d384-4e21-829a-cb2ab5f70636","Type":"ContainerDied","Data":"b0dedbc364e62b0ccf1ea854179097f91aa1a0e4f65f9f415b147837c6170248"}
Oct 30 19:55:48.910659 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.904936    4823 scope.go:122] "RemoveContainer" containerID="365876976e2dfe8654db03f62ce198063b7f3908ad4d3872739fa3fd2d61e9a9"
Oct 30 19:55:48.910659 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.904936    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"
Oct 30 19:55:48.959689 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.959660    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"]
Oct 30 19:55:48.967222 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:48.967193    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-df345a65-openshift-default-6d84bb89bf-ddk6n"]
Oct 30 19:55:49.363252 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:49.363217    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:55:50.380260 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:50.378908    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0947f714-d384-4e21-829a-cb2ab5f70636" path="/var/lib/kubelet/pods/0947f714-d384-4e21-829a-cb2ab5f70636/volumes"
Oct 30 19:55:52.286032 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:52.285959    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:52.433978 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:52.432005    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:52.800917 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:52.779685    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:52.847889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:52.806289    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:52.882889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:52.882109    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:52.882889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:52.882579    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:55:52.894158 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:52.884001    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.507912 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.507846    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:55:53.509596 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.509560    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:55:53.512252 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.512218    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.538889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.525999    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:55:53.578341 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.541734    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.595317 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.595281    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:55:53.595603 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.595588    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:55:53.632900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.542132    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.800940 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.796986    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.876889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.852183    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:55:53.892888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.884953    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:55:53.937446 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.937413    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:55:53.963198 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.963149    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.963658 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.963627    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:53.964035 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.964016    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:55:53.964249 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.964235    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:55:53.977901 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:53.966252    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:54.422728 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:54.422682    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj" podUID="f7954659-6a11-45ba-80e1-af0684ae90a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:54.600373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:54.600315    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:54.626888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:54.618962    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:55:54.635886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:54.628566    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded"
Oct 30 19:55:54.635886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:54.628686    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:55:54.662375 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:54.662328    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:54.887900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:54.884970    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:54.887900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:54.886475    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:55:54.887900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:54.886536    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:55:54.887900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:54.886577    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:55:54.913300 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:54.913016    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:55.050042 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:54.962082    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:55:55.050042 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:54.964192    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:55.050042 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:54.964513    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:55.352946 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:55.343600    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:55.352946 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:55.343673    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc" podUID="9d0dea3a-53d6-425f-a202-506d89080874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:55.502762 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:55.502712    4823 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:55.503124 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:55.503066    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps" podUID="181fb3fa-5478-4c39-9cfe-de04f2070e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:55.590912 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:55.567146    4823 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:55.590912 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:55.567210    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf" podUID="86bda8bd-f59d-4f99-aafb-cccfbf84384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:55.625105 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:55.601083    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:55.625105 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:55.619954    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:55.634519 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:55.634464    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:55.676222 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:55.676174    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:56.083760 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:56.029211    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:56.116888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:56.048847    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:55:56.289885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:56.233216    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:56.299312 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:56.241815    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:56.300050 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:56.266075    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:56.567889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:56.564016    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:56.609887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:56.605130    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:55:56.645889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:56.620445    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:56.647162 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:56.647120    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:55:56.702405 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:56.701588    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:57.005529 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.005492    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"]
Oct 30 19:55:57.013548 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.013511    4823 cpu_manager.go:414] "containerMap: removing container" podUID="0947f714-d384-4e21-829a-cb2ab5f70636" containerName="istio-proxy"
Oct 30 19:55:57.048090 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.048058    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0947f714-d384-4e21-829a-cb2ab5f70636" containerName="istio-proxy"
Oct 30 19:55:57.048463 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.048433    4823 cpu_manager.go:414] "containerMap: removing container" podUID="0947f714-d384-4e21-829a-cb2ab5f70636" containerName="istio-proxy"
Oct 30 19:55:57.048848 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.048830    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0947f714-d384-4e21-829a-cb2ab5f70636" containerName="istio-proxy"
Oct 30 19:55:57.049077 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.049063    4823 cpu_manager.go:414] "containerMap: removing container" podUID="966ec1a3-190e-4018-b40c-2055e6aefc00" containerName="revoke-keys"
Oct 30 19:55:57.049354 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.049338    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="966ec1a3-190e-4018-b40c-2055e6aefc00" containerName="revoke-keys"
Oct 30 19:55:57.076176 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.075978    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="0947f714-d384-4e21-829a-cb2ab5f70636" containerName="istio-proxy"
Oct 30 19:55:57.076408 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.076391    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="0947f714-d384-4e21-829a-cb2ab5f70636" containerName="istio-proxy"
Oct 30 19:55:57.076701 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.076685    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="966ec1a3-190e-4018-b40c-2055e6aefc00" containerName="revoke-keys"
Oct 30 19:55:57.124034 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.123983    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:57.196876 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.137370    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:57.301992 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.301881    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:57.310132 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.303424    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:57.511891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.365228    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:55:57.511891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.454554    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:57.511891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.455596    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"]
Oct 30 19:55:57.537119 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.535134    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-a8de3b-openshift-default-dockercfg-mfncj\""
Oct 30 19:55:57.608672 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.608623    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:57.652132 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.652070    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:57.740896 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.683919    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:57.791136 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.791098    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-envoy\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:57.791388 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.791365    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w89wb\" (UniqueName: \"kubernetes.io/projected/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-kube-api-access-w89wb\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:57.791614 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.791593    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istiod-ca-cert\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:57.791785 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.791766    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-credential-socket\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:57.792092 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.792071    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-workload-certs\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:57.822011 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.812399    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-workload-socket\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:57.844100 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.844064    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-token\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:57.852024 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.851986    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-data\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:57.864921 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.862394    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-podinfo\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:57.870112 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:57.870072    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:58.114324 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.114270    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-envoy\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.207368 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.207332    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w89wb\" (UniqueName: \"kubernetes.io/projected/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-kube-api-access-w89wb\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.363102 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.363067    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istiod-ca-cert\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.410625 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.410531    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-credential-socket\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.411023 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.410996    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-workload-certs\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.411497 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.411472    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-workload-socket\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.460888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.412548    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-workload-socket\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.470380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.200190    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 19:55:58.487884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.364126    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istiod-ca-cert\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.488347 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.386902    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:58.489221 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.411990    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-workload-certs\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.489448 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.200153    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:55:58.503905 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.412126    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-credential-socket\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.503905 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.200227    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:58.521540 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.331009    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:55:58.521540 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.412605    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-token\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.521540 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.505522    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-data\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.521540 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.505575    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-podinfo\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.521540 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.510297    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-data\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.521540 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.331052    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:55:58.521540 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.468670    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-envoy\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.552736 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.552700    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-podinfo\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.876950 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.612952    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:55:58.876950 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.613732    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-token\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:58.876950 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.817122    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:58.881291 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.817170    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:55:58.913624 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.860044    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:58.913624 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:58.907619    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:59.057908 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:59.041904    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w89wb\" (UniqueName: \"kubernetes.io/projected/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-kube-api-access-w89wb\") pod \"e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") " pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:59.109357 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:59.109321    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:55:59.501438 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:59.489334    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:55:59.501438 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:59.490349    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:55:59.501438 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:59.490694    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:55:59.501438 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:59.500589    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:59.538886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:59.537560    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:59.538886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:59.537981    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:59.623920 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:59.607962    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:59.739152 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:59.739034    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:59.934466 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:59.906393    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:59.950629 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:59.950001    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:59.954257 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:59.933443    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:55:59.955496 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:55:59.907981    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:00.019035 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:00.018839    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:56:00.022230 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:00.022205    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:56:00.098985 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:00.098957    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:56:00.101142 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:00.100950    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:56:00.162119 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:00.162079    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:56:00.215290 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:00.215260    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:56:00.215690 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:00.215554    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:56:00.337797 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:56:00.337751    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7862ef1c_a91c_40f6_abaf_d5eba65b67c5.slice/crio-c76d7cc73b77119a7d9c8d0be4e5ef41e2fb00f3e0d32845def62983068446f8 WatchSource:0}: Error finding container c76d7cc73b77119a7d9c8d0be4e5ef41e2fb00f3e0d32845def62983068446f8: Status 404 returned error can't find the container with id c76d7cc73b77119a7d9c8d0be4e5ef41e2fb00f3e0d32845def62983068446f8
Oct 30 19:56:00.338104 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:00.338079    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"]
Oct 30 19:56:00.341038 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:00.340997    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:56:00.341163 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:00.341089    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:56:00.341163 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:00.341132    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:56:00.843149 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:00.843102    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95" event={"ID":"7862ef1c-a91c-40f6-abaf-d5eba65b67c5","Type":"ContainerStarted","Data":"af24e8b669cd122de5bdd2c3880f7b06e4a30aefdfb0f753d9590bfb47ee3c42"}
Oct 30 19:56:00.843356 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:00.843177    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95" event={"ID":"7862ef1c-a91c-40f6-abaf-d5eba65b67c5","Type":"ContainerStarted","Data":"c76d7cc73b77119a7d9c8d0be4e5ef41e2fb00f3e0d32845def62983068446f8"}
Oct 30 19:56:00.902448 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:00.902381    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95" podStartSLOduration=5.9023598360000005 podStartE2EDuration="5.902359836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:00.901037935 +0000 UTC m=+1560.793652260" watchObservedRunningTime="2026-10-30 19:56:00.902359836 +0000 UTC m=+1560.794974161"
Oct 30 19:56:01.118884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:01.116020    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:56:01.131887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:01.129272    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:56:01.878891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:01.878298    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:56:01.882315 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:01.882286    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:56:02.019501 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:56:02.019470    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:56:07.799177 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:07.799090    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"]
Oct 30 19:56:07.808784 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:07.808750    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea210528-vkwm5"]
Oct 30 19:56:08.289698 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:08.289657    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208b36f6-c28f-4e09-a03f-19cffd9a695f" path="/var/lib/kubelet/pods/208b36f6-c28f-4e09-a03f-19cffd9a695f/volumes"
Oct 30 19:56:08.976747 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:08.976711    4823 generic.go:356] "Generic (PLEG): container finished" podID="9e5706df-826d-4682-a77b-cbd1ae21bdef" containerID="877d6784fc31f6d448670623b8d5e7adf39453fe5e7df6b2ad2f0d228b6181dd" exitCode=0
Oct 30 19:56:08.977162 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:08.976784    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9" event={"ID":"9e5706df-826d-4682-a77b-cbd1ae21bdef","Type":"ContainerDied","Data":"877d6784fc31f6d448670623b8d5e7adf39453fe5e7df6b2ad2f0d228b6181dd"}
Oct 30 19:56:08.977397 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:08.977371    4823 scope.go:122] "RemoveContainer" containerID="877d6784fc31f6d448670623b8d5e7adf39453fe5e7df6b2ad2f0d228b6181dd"
Oct 30 19:56:08.979324 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:08.979288    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:56:08.979427 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:08.979365    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:56:08.979427 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:08.979396    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:56:09.353752 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:09.353654    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"]
Oct 30 19:56:09.586887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:09.584018    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:56:09.993657 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:09.993595    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9" event={"ID":"9e5706df-826d-4682-a77b-cbd1ae21bdef","Type":"ContainerStarted","Data":"e9b57c4c1923cc71b96574e9c36d9d7becedaeebb5ac8b62d5bf8b8a09ab3669"}
Oct 30 19:56:09.994250 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:09.994037    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9" podUID="9e5706df-826d-4682-a77b-cbd1ae21bdef" containerName="istio-proxy" containerID="cri-o://e9b57c4c1923cc71b96574e9c36d9d7becedaeebb5ac8b62d5bf8b8a09ab3669" gracePeriod=30
Oct 30 19:56:10.678888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:10.675849    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"]
Oct 30 19:56:10.817367 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:10.815002    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"]
Oct 30 19:56:10.832904 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:10.832850    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:10.840413 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:10.840365    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d07cb404-scnwb"]
Oct 30 19:56:10.847606 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:10.847562    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"]
Oct 30 19:56:10.864774 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:10.862385    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e7f2a520-openshift-default-dockercfg-ww5fm\""
Oct 30 19:56:10.904971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:10.903375    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-podinfo\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:10.904971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:10.903517    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-data\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:10.904971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:10.903628    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-workload-certs\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:10.904971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:10.903661    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-workload-socket\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:10.904971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:10.903702    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-credential-socket\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:10.904971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:10.903784    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-envoy\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:10.904971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:10.903887    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istiod-ca-cert\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:10.904971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:10.903943    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmhhv\" (UniqueName: \"kubernetes.io/projected/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-kube-api-access-fmhhv\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:10.904971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:10.904007    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-token\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.073938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.048410    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-workload-certs\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.073938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.048468    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-workload-socket\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.073938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.048516    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-credential-socket\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.073938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.048594    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-envoy\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.073938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.048682    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istiod-ca-cert\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.073938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.048736    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmhhv\" (UniqueName: \"kubernetes.io/projected/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-kube-api-access-fmhhv\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.073938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.048793    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-token\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.073938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.048886    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-podinfo\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.073938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.049010    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-data\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.080759 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.080710    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-workload-certs\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.104888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.081647    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-data\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.105144 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.086407    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-workload-socket\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.105291 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.090027    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-credential-socket\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.105610 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.105581    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istiod-ca-cert\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.124368 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.120805    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-token\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.130242 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.127126    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-envoy\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.130242 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.127779    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmhhv\" (UniqueName: \"kubernetes.io/projected/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-kube-api-access-fmhhv\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.130242 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.130059    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-podinfo\") pod \"e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") " pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.347323 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.347211    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:11.745758 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:56:11.745719    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4253cfc3_c0b6_415b_919d_1fd829ab9ae8.slice/crio-ef49d622e12835f8b700cc3d67b2367274c06dd68c7293ddc90b86c94029e333 WatchSource:0}: Error finding container ef49d622e12835f8b700cc3d67b2367274c06dd68c7293ddc90b86c94029e333: Status 404 returned error can't find the container with id ef49d622e12835f8b700cc3d67b2367274c06dd68c7293ddc90b86c94029e333
Oct 30 19:56:11.746399 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.746374    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"]
Oct 30 19:56:11.749024 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.748989    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:56:11.749123 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.749060    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:56:11.749123 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:11.749089    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:56:12.093051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:12.092941    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8" event={"ID":"4253cfc3-c0b6-415b-919d-1fd829ab9ae8","Type":"ContainerStarted","Data":"11ea93849dd73719e5982bc3e81b0efe6ec06b0ca55c43c18be99f27be5bec24"}
Oct 30 19:56:12.093051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:12.093011    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8" event={"ID":"4253cfc3-c0b6-415b-919d-1fd829ab9ae8","Type":"ContainerStarted","Data":"ef49d622e12835f8b700cc3d67b2367274c06dd68c7293ddc90b86c94029e333"}
Oct 30 19:56:12.129370 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:12.129317    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8" podStartSLOduration=2.129300963 podStartE2EDuration="2.129300963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:12.125257498 +0000 UTC m=+1572.017871821" watchObservedRunningTime="2026-10-30 19:56:12.129300963 +0000 UTC m=+1572.021915287"
Oct 30 19:56:12.331902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:12.329639    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="966ec1a3-190e-4018-b40c-2055e6aefc00" path="/var/lib/kubelet/pods/966ec1a3-190e-4018-b40c-2055e6aefc00/volumes"
Oct 30 19:56:12.347624 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:12.347547    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:12.354529 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:12.354497    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:13.102412 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:13.102380    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:13.103444 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:13.103423    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:56:15.124164 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.124114    4823 generic.go:356] "Generic (PLEG): container finished" podID="9e5706df-826d-4682-a77b-cbd1ae21bdef" containerID="e9b57c4c1923cc71b96574e9c36d9d7becedaeebb5ac8b62d5bf8b8a09ab3669" exitCode=0
Oct 30 19:56:15.124634 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.124195    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9" event={"ID":"9e5706df-826d-4682-a77b-cbd1ae21bdef","Type":"ContainerDied","Data":"e9b57c4c1923cc71b96574e9c36d9d7becedaeebb5ac8b62d5bf8b8a09ab3669"}
Oct 30 19:56:15.124634 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.124275    4823 scope.go:122] "RemoveContainer" containerID="877d6784fc31f6d448670623b8d5e7adf39453fe5e7df6b2ad2f0d228b6181dd"
Oct 30 19:56:15.748851 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.748388    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:56:15.936344 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.936298    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5706df-826d-4682-a77b-cbd1ae21bdef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5706df-826d-4682-a77b-cbd1ae21bdef-istiod-ca-cert\") pod \"9e5706df-826d-4682-a77b-cbd1ae21bdef\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") "
Oct 30 19:56:15.936539 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.936366    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-workload-certs\") pod \"9e5706df-826d-4682-a77b-cbd1ae21bdef\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") "
Oct 30 19:56:15.936539 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.936488    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-workload-socket\") pod \"9e5706df-826d-4682-a77b-cbd1ae21bdef\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") "
Oct 30 19:56:15.936643 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.936547    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5706df-826d-4682-a77b-cbd1ae21bdef-kube-api-access-sjlmf\" (UniqueName: \"kubernetes.io/projected/9e5706df-826d-4682-a77b-cbd1ae21bdef-kube-api-access-sjlmf\") pod \"9e5706df-826d-4682-a77b-cbd1ae21bdef\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") "
Oct 30 19:56:15.936715 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.936644    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-workload-certs" pod "9e5706df-826d-4682-a77b-cbd1ae21bdef" (UID: "9e5706df-826d-4682-a77b-cbd1ae21bdef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:15.936715 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.936693    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-data\") pod \"9e5706df-826d-4682-a77b-cbd1ae21bdef\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") "
Oct 30 19:56:15.936828 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.936766    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-workload-socket" pod "9e5706df-826d-4682-a77b-cbd1ae21bdef" (UID: "9e5706df-826d-4682-a77b-cbd1ae21bdef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:15.936828 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.936783    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-token\" (UniqueName: \"kubernetes.io/projected/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-token\") pod \"9e5706df-826d-4682-a77b-cbd1ae21bdef\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") "
Oct 30 19:56:15.936828 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.936804    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5706df-826d-4682-a77b-cbd1ae21bdef-istiod-ca-cert" pod "9e5706df-826d-4682-a77b-cbd1ae21bdef" (UID: "9e5706df-826d-4682-a77b-cbd1ae21bdef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:15.936828 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.936825    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-credential-socket\") pod \"9e5706df-826d-4682-a77b-cbd1ae21bdef\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") "
Oct 30 19:56:15.937106 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.936856    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-envoy\") pod \"9e5706df-826d-4682-a77b-cbd1ae21bdef\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") "
Oct 30 19:56:15.937106 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.937001    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-data" pod "9e5706df-826d-4682-a77b-cbd1ae21bdef" (UID: "9e5706df-826d-4682-a77b-cbd1ae21bdef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:15.937106 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.937065    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-credential-socket" pod "9e5706df-826d-4682-a77b-cbd1ae21bdef" (UID: "9e5706df-826d-4682-a77b-cbd1ae21bdef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:15.937106 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.937102    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-podinfo\") pod \"9e5706df-826d-4682-a77b-cbd1ae21bdef\" (UID: \"9e5706df-826d-4682-a77b-cbd1ae21bdef\") "
Oct 30 19:56:15.937774 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.937744    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:15.937774 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.937767    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:15.937965 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.937780    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:15.937965 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.937794    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5706df-826d-4682-a77b-cbd1ae21bdef-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:15.937965 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.937806    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:15.939324 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.939299    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-token" pod "9e5706df-826d-4682-a77b-cbd1ae21bdef" (UID: "9e5706df-826d-4682-a77b-cbd1ae21bdef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:15.939488 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.939469    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-envoy" pod "9e5706df-826d-4682-a77b-cbd1ae21bdef" (UID: "9e5706df-826d-4682-a77b-cbd1ae21bdef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:15.939553 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.939489    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-podinfo" pod "9e5706df-826d-4682-a77b-cbd1ae21bdef" (UID: "9e5706df-826d-4682-a77b-cbd1ae21bdef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:56:15.939591 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:15.939552    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5706df-826d-4682-a77b-cbd1ae21bdef-kube-api-access-sjlmf" pod "9e5706df-826d-4682-a77b-cbd1ae21bdef" (UID: "9e5706df-826d-4682-a77b-cbd1ae21bdef"). InnerVolumeSpecName "kube-api-access-sjlmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:16.039065 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:16.039025    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjlmf\" (UniqueName: \"kubernetes.io/projected/9e5706df-826d-4682-a77b-cbd1ae21bdef-kube-api-access-sjlmf\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:16.039065 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:16.039061    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:16.039065 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:16.039073    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:16.039326 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:16.039082    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5706df-826d-4682-a77b-cbd1ae21bdef-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:16.135645 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:16.135611    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"
Oct 30 19:56:16.135645 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:16.135629    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9" event={"ID":"9e5706df-826d-4682-a77b-cbd1ae21bdef","Type":"ContainerDied","Data":"227bff5f0b7790a6685d5bb2f7bdfa6aec979df14683effa88de1a3513a7c514"}
Oct 30 19:56:16.136229 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:16.135689    4823 scope.go:122] "RemoveContainer" containerID="e9b57c4c1923cc71b96574e9c36d9d7becedaeebb5ac8b62d5bf8b8a09ab3669"
Oct 30 19:56:16.170257 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:16.170228    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"]
Oct 30 19:56:16.179352 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:16.179320    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ea210528-openshift-default-5999c9584f-mdlx9"]
Oct 30 19:56:16.281651 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:16.281571    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5706df-826d-4682-a77b-cbd1ae21bdef" path="/var/lib/kubelet/pods/9e5706df-826d-4682-a77b-cbd1ae21bdef/volumes"
Oct 30 19:56:21.052740 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.052705    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"]
Oct 30 19:56:21.053526 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.053509    4823 cpu_manager.go:414] "containerMap: removing container" podUID="9e5706df-826d-4682-a77b-cbd1ae21bdef" containerName="istio-proxy"
Oct 30 19:56:21.053582 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.053528    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5706df-826d-4682-a77b-cbd1ae21bdef" containerName="istio-proxy"
Oct 30 19:56:21.053721 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.053710    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5706df-826d-4682-a77b-cbd1ae21bdef" containerName="istio-proxy"
Oct 30 19:56:21.053757 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.053724    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5706df-826d-4682-a77b-cbd1ae21bdef" containerName="istio-proxy"
Oct 30 19:56:21.056682 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.056665    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"
Oct 30 19:56:21.062154 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.062122    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:56:21.062154 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.062130    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:56:21.062395 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.062135    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:56:21.062586 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.062557    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:56:21.070824 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.070789    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"]
Oct 30 19:56:21.153524 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.153484    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fnjj\" (UniqueName: \"kubernetes.io/projected/0a78c503-c362-4f0d-982a-9894bf6650dd-kube-api-access-4fnjj\") pod \"maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf\" (UID: \"0a78c503-c362-4f0d-982a-9894bf6650dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"
Oct 30 19:56:21.153709 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.153675    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a78c503-c362-4f0d-982a-9894bf6650dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf\" (UID: \"0a78c503-c362-4f0d-982a-9894bf6650dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"
Oct 30 19:56:21.153758 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.153736    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a78c503-c362-4f0d-982a-9894bf6650dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf\" (UID: \"0a78c503-c362-4f0d-982a-9894bf6650dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"
Oct 30 19:56:21.255303 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.255258    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fnjj\" (UniqueName: \"kubernetes.io/projected/0a78c503-c362-4f0d-982a-9894bf6650dd-kube-api-access-4fnjj\") pod \"maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf\" (UID: \"0a78c503-c362-4f0d-982a-9894bf6650dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"
Oct 30 19:56:21.255518 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.255380    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a78c503-c362-4f0d-982a-9894bf6650dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf\" (UID: \"0a78c503-c362-4f0d-982a-9894bf6650dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"
Oct 30 19:56:21.255518 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.255433    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a78c503-c362-4f0d-982a-9894bf6650dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf\" (UID: \"0a78c503-c362-4f0d-982a-9894bf6650dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"
Oct 30 19:56:21.257326 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.256626    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a78c503-c362-4f0d-982a-9894bf6650dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf\" (UID: \"0a78c503-c362-4f0d-982a-9894bf6650dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"
Oct 30 19:56:21.259174 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.259112    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a78c503-c362-4f0d-982a-9894bf6650dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf\" (UID: \"0a78c503-c362-4f0d-982a-9894bf6650dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"
Oct 30 19:56:21.272058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.272025    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fnjj\" (UniqueName: \"kubernetes.io/projected/0a78c503-c362-4f0d-982a-9894bf6650dd-kube-api-access-4fnjj\") pod \"maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf\" (UID: \"0a78c503-c362-4f0d-982a-9894bf6650dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"
Oct 30 19:56:21.383295 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.383263    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"
Oct 30 19:56:21.678921 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:56:21.678883    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a78c503_c362_4f0d_982a_9894bf6650dd.slice/crio-128323b525db04c33978181ee65f8838de449239b7dcc7c01253e32f8070284b WatchSource:0}: Error finding container 128323b525db04c33978181ee65f8838de449239b7dcc7c01253e32f8070284b: Status 404 returned error can't find the container with id 128323b525db04c33978181ee65f8838de449239b7dcc7c01253e32f8070284b
Oct 30 19:56:21.680538 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:21.680507    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"]
Oct 30 19:56:22.207062 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:22.207022    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf" event={"ID":"0a78c503-c362-4f0d-982a-9894bf6650dd","Type":"ContainerStarted","Data":"5b969b5f79707c83e4b70ddcc05702c3224fb7444270c4176f2068b68121c061"}
Oct 30 19:56:22.207539 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:22.207083    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf" event={"ID":"0a78c503-c362-4f0d-982a-9894bf6650dd","Type":"ContainerStarted","Data":"128323b525db04c33978181ee65f8838de449239b7dcc7c01253e32f8070284b"}
Oct 30 19:56:22.225814 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:22.225751    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf" podStartSLOduration=1.225734121 podStartE2EDuration="1.225734121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:22.224215934 +0000 UTC m=+1582.116830258" watchObservedRunningTime="2026-10-30 19:56:22.225734121 +0000 UTC m=+1582.118348446"
Oct 30 19:56:23.218760 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:23.218724    4823 generic.go:356] "Generic (PLEG): container finished" podID="0a78c503-c362-4f0d-982a-9894bf6650dd" containerID="5b969b5f79707c83e4b70ddcc05702c3224fb7444270c4176f2068b68121c061" exitCode=0
Oct 30 19:56:23.219214 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:23.218807    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf" event={"ID":"0a78c503-c362-4f0d-982a-9894bf6650dd","Type":"ContainerDied","Data":"5b969b5f79707c83e4b70ddcc05702c3224fb7444270c4176f2068b68121c061"}
Oct 30 19:56:24.497144 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:24.497073    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"
Oct 30 19:56:24.590773 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:24.590737    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a78c503-c362-4f0d-982a-9894bf6650dd-kube-api-access-4fnjj\" (UniqueName: \"kubernetes.io/projected/0a78c503-c362-4f0d-982a-9894bf6650dd-kube-api-access-4fnjj\") pod \"0a78c503-c362-4f0d-982a-9894bf6650dd\" (UID: \"0a78c503-c362-4f0d-982a-9894bf6650dd\") "
Oct 30 19:56:24.590994 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:24.590830    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a78c503-c362-4f0d-982a-9894bf6650dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a78c503-c362-4f0d-982a-9894bf6650dd-maas-api-xks-ca\") pod \"0a78c503-c362-4f0d-982a-9894bf6650dd\" (UID: \"0a78c503-c362-4f0d-982a-9894bf6650dd\") "
Oct 30 19:56:24.591167 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:24.591144    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a78c503-c362-4f0d-982a-9894bf6650dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a78c503-c362-4f0d-982a-9894bf6650dd-maas-api-openshift-ca\") pod \"0a78c503-c362-4f0d-982a-9894bf6650dd\" (UID: \"0a78c503-c362-4f0d-982a-9894bf6650dd\") "
Oct 30 19:56:24.591652 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:24.591620    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a78c503-c362-4f0d-982a-9894bf6650dd-maas-api-openshift-ca" pod "0a78c503-c362-4f0d-982a-9894bf6650dd" (UID: "0a78c503-c362-4f0d-982a-9894bf6650dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:24.592160 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:24.592138    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a78c503-c362-4f0d-982a-9894bf6650dd-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:24.593137 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:24.593108    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a78c503-c362-4f0d-982a-9894bf6650dd-maas-api-xks-ca" pod "0a78c503-c362-4f0d-982a-9894bf6650dd" (UID: "0a78c503-c362-4f0d-982a-9894bf6650dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:24.593271 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:24.593247    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a78c503-c362-4f0d-982a-9894bf6650dd-kube-api-access-4fnjj" pod "0a78c503-c362-4f0d-982a-9894bf6650dd" (UID: "0a78c503-c362-4f0d-982a-9894bf6650dd"). InnerVolumeSpecName "kube-api-access-4fnjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:24.704245 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:24.704210    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fnjj\" (UniqueName: \"kubernetes.io/projected/0a78c503-c362-4f0d-982a-9894bf6650dd-kube-api-access-4fnjj\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:24.704457 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:24.704444    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a78c503-c362-4f0d-982a-9894bf6650dd-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:25.249107 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:25.249075    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"
Oct 30 19:56:25.249258 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:25.249103    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf" event={"ID":"0a78c503-c362-4f0d-982a-9894bf6650dd","Type":"ContainerDied","Data":"128323b525db04c33978181ee65f8838de449239b7dcc7c01253e32f8070284b"}
Oct 30 19:56:25.249258 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:25.249152    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="128323b525db04c33978181ee65f8838de449239b7dcc7c01253e32f8070284b"
Oct 30 19:56:30.889887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:30.888956    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:30.981350 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:30.974102    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:30.981350 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:30.975946    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:56:31.128213 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:31.128159    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:56:31.248946 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:31.166032    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:31.525947 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:31.498985    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:31.715889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:31.690680    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:56:31.715889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:31.692083    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:31.715889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:31.704211    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:56:31.715889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:31.713274    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:31.894931 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:31.894447    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:31.948527 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:31.912132    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:56:31.986292 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:31.983248    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:56:32.005220 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:31.995142    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:56:32.221939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:32.196737    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:32.312887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:32.312090    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:56:32.716079 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:32.716025    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:32.718891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:32.718831    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:32.733557 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:32.719657    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:32.734826 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:32.720438    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:32.826215 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:32.817743    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:32.899020 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:32.898968    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:32.982940 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:32.972882    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:56:32.982940 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:32.972961    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:56:32.982940 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:32.973008    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:56:32.982940 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:32.973050    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:56:32.982940 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:32.973592    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:56:32.982940 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:32.973642    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:56:32.999248 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:32.999207    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:33.015831 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:33.005996    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:33.354442 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:33.354320    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:33.484619 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:33.354782    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:56:33.501454 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:33.381324    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:33.719350 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:33.719302    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:33.741891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:33.741189    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:56:33.750892 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:33.750560    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:56:33.781348 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:33.759591    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:56:33.900439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:33.899947    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:34.031946 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:34.014306    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:56:34.049893 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:34.043245    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:56:34.440085 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:34.440044    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7fhfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:56:34.440389 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:34.440354    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw" podUID="3eb29b61-5760-4a85-8573-a3205e66e833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:34.545936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:34.529986    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:34.553642 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:34.552534    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:56:34.576763 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:34.575695    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:34.610901 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:34.608485    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-4zvvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:56:34.610901 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:34.608561    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg" podUID="11656a86-294a-4cd1-a29c-6f8d02a39c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:34.610901 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:56:34.609223    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:56:34.641932 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:34.641105    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:56:34.745111 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:34.745006    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:34.746751 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:34.746715    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:56:34.775303 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:34.775252    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:56:34.840052 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:34.840000    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:34.923051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:34.910454    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:35.065934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:35.064938    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:56:35.090734 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:35.078679    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:35.697971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:35.658079    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:56:35.699312 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:35.698541    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:56:35.751000 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:35.746420    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:35.776891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:35.751496    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:35.853890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:35.776011    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:56:35.854973 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:35.854915    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:56:35.924910 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:35.924847    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:36.083947 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.066055    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:36.228527 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.093744    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:56:36.616888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.564289    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:56:36.665054 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.638115    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"]
Oct 30 19:56:36.692150 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.692119    4823 cpu_manager.go:414] "containerMap: removing container" podUID="0a78c503-c362-4f0d-982a-9894bf6650dd" containerName="revoke-keys"
Oct 30 19:56:36.692376 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.692361    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a78c503-c362-4f0d-982a-9894bf6650dd" containerName="revoke-keys"
Oct 30 19:56:36.692530 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.692519    4823 cpu_manager.go:414] "containerMap: removing container" podUID="9e5706df-826d-4682-a77b-cbd1ae21bdef" containerName="istio-proxy"
Oct 30 19:56:36.692620 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.692610    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5706df-826d-4682-a77b-cbd1ae21bdef" containerName="istio-proxy"
Oct 30 19:56:36.692970 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.692953    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a78c503-c362-4f0d-982a-9894bf6650dd" containerName="revoke-keys"
Oct 30 19:56:36.711529 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.711478    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:56:36.711764 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.711563    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"
Oct 30 19:56:36.711915 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.711896    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:56:36.712052 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.712039    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"]
Oct 30 19:56:36.715321 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.715296    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:56:36.715615 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.715599    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:56:36.715790 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.715775    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:56:36.716008 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.715991    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:56:36.719211 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.719180    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"]
Oct 30 19:56:36.719211 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.719215    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"]
Oct 30 19:56:36.719402 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.719262    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:56:36.719402 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.719289    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:56:36.719402 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.719379    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:36.719554 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.719438    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:56:36.719693 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.719676    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:56:36.719775 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.719718    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:56:36.719775 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.719751    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:56:36.722743 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.722721    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-f5d176-openshift-default-dockercfg-ff257\""
Oct 30 19:56:36.801528 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.801478    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afd2b7d3-be2d-446d-9824-6d89903b0938-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a81713d6-xjpj7\" (UID: \"afd2b7d3-be2d-446d-9824-6d89903b0938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"
Oct 30 19:56:36.801748 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.801636    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afd2b7d3-be2d-446d-9824-6d89903b0938-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a81713d6-xjpj7\" (UID: \"afd2b7d3-be2d-446d-9824-6d89903b0938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"
Oct 30 19:56:36.801828 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.801777    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd894\" (UniqueName: \"kubernetes.io/projected/afd2b7d3-be2d-446d-9824-6d89903b0938-kube-api-access-fd894\") pod \"maas-api-revoke-sub-a81713d6-xjpj7\" (UID: \"afd2b7d3-be2d-446d-9824-6d89903b0938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"
Oct 30 19:56:36.919305 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.914875    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-podinfo\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:36.919305 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.914973    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afd2b7d3-be2d-446d-9824-6d89903b0938-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a81713d6-xjpj7\" (UID: \"afd2b7d3-be2d-446d-9824-6d89903b0938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"
Oct 30 19:56:36.919305 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.915091    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-data\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:36.919305 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.915170    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-workload-certs\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:36.919305 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.915222    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afd2b7d3-be2d-446d-9824-6d89903b0938-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a81713d6-xjpj7\" (UID: \"afd2b7d3-be2d-446d-9824-6d89903b0938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"
Oct 30 19:56:36.919305 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.915269    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-envoy\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:36.919305 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.915343    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istiod-ca-cert\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:36.919305 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.915400    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-credential-socket\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:36.919305 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.915477    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd894\" (UniqueName: \"kubernetes.io/projected/afd2b7d3-be2d-446d-9824-6d89903b0938-kube-api-access-fd894\") pod \"maas-api-revoke-sub-a81713d6-xjpj7\" (UID: \"afd2b7d3-be2d-446d-9824-6d89903b0938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"
Oct 30 19:56:36.919305 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.915571    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-token\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:36.919305 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.915645    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdrmr\" (UniqueName: \"kubernetes.io/projected/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-kube-api-access-kdrmr\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:36.919305 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.915724    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-workload-socket\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:36.920081 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.920047    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afd2b7d3-be2d-446d-9824-6d89903b0938-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a81713d6-xjpj7\" (UID: \"afd2b7d3-be2d-446d-9824-6d89903b0938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"
Oct 30 19:56:36.922328 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.922285    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afd2b7d3-be2d-446d-9824-6d89903b0938-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a81713d6-xjpj7\" (UID: \"afd2b7d3-be2d-446d-9824-6d89903b0938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"
Oct 30 19:56:36.936827 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:36.936795    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd894\" (UniqueName: \"kubernetes.io/projected/afd2b7d3-be2d-446d-9824-6d89903b0938-kube-api-access-fd894\") pod \"maas-api-revoke-sub-a81713d6-xjpj7\" (UID: \"afd2b7d3-be2d-446d-9824-6d89903b0938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"
Oct 30 19:56:37.017829 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.017778    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-podinfo\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.018054 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.017913    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-data\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.018054 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.017966    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-workload-certs\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.018054 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.018002    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-envoy\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.018226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.018053    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istiod-ca-cert\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.018226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.018099    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-credential-socket\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.018226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.018218    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-token\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.018382 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.018266    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdrmr\" (UniqueName: \"kubernetes.io/projected/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-kube-api-access-kdrmr\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.018382 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.018316    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-workload-socket\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.018493 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.018397    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-data\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.018576 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.018552    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-workload-certs\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.018928 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.018846    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-workload-socket\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.019072 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.018984    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-credential-socket\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.019138 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.019077    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istiod-ca-cert\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.021252 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.021227    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-envoy\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.021388 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.021318    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-podinfo\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.030508 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.030472    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"
Oct 30 19:56:37.047572 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.047544    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdrmr\" (UniqueName: \"kubernetes.io/projected/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-kube-api-access-kdrmr\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.047816 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.047788    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-token\") pod \"e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") " pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.100944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.089282    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:37.602697 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:56:37.602669    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd2b7d3_be2d_446d_9824_6d89903b0938.slice/crio-a20c29635e921f2c33b6c9d23c5abd47c05ce1c7419cc2c3eb4e7e115ca99c30 WatchSource:0}: Error finding container a20c29635e921f2c33b6c9d23c5abd47c05ce1c7419cc2c3eb4e7e115ca99c30: Status 404 returned error can't find the container with id a20c29635e921f2c33b6c9d23c5abd47c05ce1c7419cc2c3eb4e7e115ca99c30
Oct 30 19:56:37.604008 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.603980    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"]
Oct 30 19:56:37.653889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.653833    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"]
Oct 30 19:56:37.664854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.664054    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:56:37.664854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.664153    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:56:37.664854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:37.664198    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:56:38.082466 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:38.081498    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"]
Oct 30 19:56:38.092927 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:38.092835    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e5e7a13-2wdv2"]
Oct 30 19:56:38.294780 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:38.294742    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86625a00-2726-4009-a7a0-456e07604687" path="/var/lib/kubelet/pods/86625a00-2726-4009-a7a0-456e07604687/volumes"
Oct 30 19:56:38.597772 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:38.597732    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7" event={"ID":"afd2b7d3-be2d-446d-9824-6d89903b0938","Type":"ContainerStarted","Data":"71f947678c3067af7b54099edaad79c2492caa65ddb65db205e7ba17317d34b6"}
Oct 30 19:56:38.597976 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:38.597803    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7" event={"ID":"afd2b7d3-be2d-446d-9824-6d89903b0938","Type":"ContainerStarted","Data":"a20c29635e921f2c33b6c9d23c5abd47c05ce1c7419cc2c3eb4e7e115ca99c30"}
Oct 30 19:56:38.599928 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:38.599897    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn" event={"ID":"cfe9f1d8-aef1-41cd-b586-c0c073d9b574","Type":"ContainerStarted","Data":"6d7549911361640349a52a5388f57952e449b8f0a495191422cd0c13a9c248d1"}
Oct 30 19:56:38.600063 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:38.599945    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn" event={"ID":"cfe9f1d8-aef1-41cd-b586-c0c073d9b574","Type":"ContainerStarted","Data":"a58c914dea42cc2fdca063c06ae17957ac61389a7f45c663d09d189260174d5c"}
Oct 30 19:56:38.639397 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:38.639344    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7" podStartSLOduration=4.639329505 podStartE2EDuration="4.639329505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:38.633708847 +0000 UTC m=+1598.526323170" watchObservedRunningTime="2026-10-30 19:56:38.639329505 +0000 UTC m=+1598.531943825"
Oct 30 19:56:38.681492 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:38.681439    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn" podStartSLOduration=4.681425783 podStartE2EDuration="4.681425783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:38.677137024 +0000 UTC m=+1598.569751350" watchObservedRunningTime="2026-10-30 19:56:38.681425783 +0000 UTC m=+1598.574040107"
Oct 30 19:56:39.090236 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:39.090202    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:39.096089 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:39.096058    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:39.689931 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:39.687155    4823 generic.go:356] "Generic (PLEG): container finished" podID="afd2b7d3-be2d-446d-9824-6d89903b0938" containerID="71f947678c3067af7b54099edaad79c2492caa65ddb65db205e7ba17317d34b6" exitCode=0
Oct 30 19:56:39.689931 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:39.687932    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7" event={"ID":"afd2b7d3-be2d-446d-9824-6d89903b0938","Type":"ContainerDied","Data":"71f947678c3067af7b54099edaad79c2492caa65ddb65db205e7ba17317d34b6"}
Oct 30 19:56:39.689931 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:39.688391    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:39.697168 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:39.696184    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:56:40.076790 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:40.076674    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"]
Oct 30 19:56:40.088785 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:40.088729    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a947d1a3-vvrdx"]
Oct 30 19:56:40.296570 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:40.296534    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5da1768-d7b5-4667-bea4-8136dd52fe63" path="/var/lib/kubelet/pods/e5da1768-d7b5-4667-bea4-8136dd52fe63/volumes"
Oct 30 19:56:41.016685 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:41.016654    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"
Oct 30 19:56:41.051675 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:41.051637    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afd2b7d3-be2d-446d-9824-6d89903b0938-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afd2b7d3-be2d-446d-9824-6d89903b0938-maas-api-xks-ca\") pod \"afd2b7d3-be2d-446d-9824-6d89903b0938\" (UID: \"afd2b7d3-be2d-446d-9824-6d89903b0938\") "
Oct 30 19:56:41.051835 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:41.051755    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd2b7d3-be2d-446d-9824-6d89903b0938-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afd2b7d3-be2d-446d-9824-6d89903b0938-maas-api-openshift-ca\") pod \"afd2b7d3-be2d-446d-9824-6d89903b0938\" (UID: \"afd2b7d3-be2d-446d-9824-6d89903b0938\") "
Oct 30 19:56:41.052007 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:41.051931    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd2b7d3-be2d-446d-9824-6d89903b0938-kube-api-access-fd894\" (UniqueName: \"kubernetes.io/projected/afd2b7d3-be2d-446d-9824-6d89903b0938-kube-api-access-fd894\") pod \"afd2b7d3-be2d-446d-9824-6d89903b0938\" (UID: \"afd2b7d3-be2d-446d-9824-6d89903b0938\") "
Oct 30 19:56:41.052212 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:41.052185    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd2b7d3-be2d-446d-9824-6d89903b0938-maas-api-openshift-ca" pod "afd2b7d3-be2d-446d-9824-6d89903b0938" (UID: "afd2b7d3-be2d-446d-9824-6d89903b0938"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:41.052788 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:41.052758    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afd2b7d3-be2d-446d-9824-6d89903b0938-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:41.054491 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:41.054452    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd2b7d3-be2d-446d-9824-6d89903b0938-maas-api-xks-ca" pod "afd2b7d3-be2d-446d-9824-6d89903b0938" (UID: "afd2b7d3-be2d-446d-9824-6d89903b0938"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:41.096821 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:41.096186    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd2b7d3-be2d-446d-9824-6d89903b0938-kube-api-access-fd894" pod "afd2b7d3-be2d-446d-9824-6d89903b0938" (UID: "afd2b7d3-be2d-446d-9824-6d89903b0938"). InnerVolumeSpecName "kube-api-access-fd894". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:41.155003 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:41.154969    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd894\" (UniqueName: \"kubernetes.io/projected/afd2b7d3-be2d-446d-9824-6d89903b0938-kube-api-access-fd894\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:41.155003 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:41.155001    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afd2b7d3-be2d-446d-9824-6d89903b0938-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:41.711366 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:41.711323    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7" event={"ID":"afd2b7d3-be2d-446d-9824-6d89903b0938","Type":"ContainerDied","Data":"a20c29635e921f2c33b6c9d23c5abd47c05ce1c7419cc2c3eb4e7e115ca99c30"}
Oct 30 19:56:41.711933 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:41.711389    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a20c29635e921f2c33b6c9d23c5abd47c05ce1c7419cc2c3eb4e7e115ca99c30"
Oct 30 19:56:41.711933 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:41.711349    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"
Oct 30 19:56:44.808920 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:44.808846    4823 generic.go:356] "Generic (PLEG): container finished" podID="622c7980-cb92-4c47-ba6a-f333e5d6256e" containerID="bc999ee4a27e237ab48f4ae52ca6746dc9659fc769d227b354c062669f573e0e" exitCode=0
Oct 30 19:56:44.809469 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:44.808926    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9" event={"ID":"622c7980-cb92-4c47-ba6a-f333e5d6256e","Type":"ContainerDied","Data":"bc999ee4a27e237ab48f4ae52ca6746dc9659fc769d227b354c062669f573e0e"}
Oct 30 19:56:44.809570 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:44.809550    4823 scope.go:122] "RemoveContainer" containerID="bc999ee4a27e237ab48f4ae52ca6746dc9659fc769d227b354c062669f573e0e"
Oct 30 19:56:44.811473 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:44.811434    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:56:44.811603 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:44.811530    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:56:44.811603 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:44.811581    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:56:44.952366 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:44.952333    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"]
Oct 30 19:56:44.961909 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:44.961083    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b69bf89d-7l7zf"]
Oct 30 19:56:45.158882 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.158821    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"]
Oct 30 19:56:45.574888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.574772    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"]
Oct 30 19:56:45.576149 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.576117    4823 cpu_manager.go:414] "containerMap: removing container" podUID="afd2b7d3-be2d-446d-9824-6d89903b0938" containerName="revoke-keys"
Oct 30 19:56:45.576149 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.576149    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd2b7d3-be2d-446d-9824-6d89903b0938" containerName="revoke-keys"
Oct 30 19:56:45.576451 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.576427    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd2b7d3-be2d-446d-9824-6d89903b0938" containerName="revoke-keys"
Oct 30 19:56:45.593944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.590723    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"
Oct 30 19:56:45.598811 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.598743    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:56:45.599030 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.598937    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:56:45.599030 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.598996    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:56:45.600063 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.600021    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:56:45.603972 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.603914    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb0e0c5-d2da-4624-a538-c872abaed25b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8\" (UID: \"ecb0e0c5-d2da-4624-a538-c872abaed25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"
Oct 30 19:56:45.604179 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.604156    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpl4f\" (UniqueName: \"kubernetes.io/projected/ecb0e0c5-d2da-4624-a538-c872abaed25b-kube-api-access-fpl4f\") pod \"maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8\" (UID: \"ecb0e0c5-d2da-4624-a538-c872abaed25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"
Oct 30 19:56:45.604622 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.604597    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb0e0c5-d2da-4624-a538-c872abaed25b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8\" (UID: \"ecb0e0c5-d2da-4624-a538-c872abaed25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"
Oct 30 19:56:45.610201 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.610150    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"]
Oct 30 19:56:45.707609 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.707561    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb0e0c5-d2da-4624-a538-c872abaed25b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8\" (UID: \"ecb0e0c5-d2da-4624-a538-c872abaed25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"
Oct 30 19:56:45.707826 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.707631    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpl4f\" (UniqueName: \"kubernetes.io/projected/ecb0e0c5-d2da-4624-a538-c872abaed25b-kube-api-access-fpl4f\") pod \"maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8\" (UID: \"ecb0e0c5-d2da-4624-a538-c872abaed25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"
Oct 30 19:56:45.707826 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.707691    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb0e0c5-d2da-4624-a538-c872abaed25b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8\" (UID: \"ecb0e0c5-d2da-4624-a538-c872abaed25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"
Oct 30 19:56:45.708512 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.708485    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb0e0c5-d2da-4624-a538-c872abaed25b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8\" (UID: \"ecb0e0c5-d2da-4624-a538-c872abaed25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"
Oct 30 19:56:45.710312 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.710275    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb0e0c5-d2da-4624-a538-c872abaed25b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8\" (UID: \"ecb0e0c5-d2da-4624-a538-c872abaed25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"
Oct 30 19:56:45.724953 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.724916    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpl4f\" (UniqueName: \"kubernetes.io/projected/ecb0e0c5-d2da-4624-a538-c872abaed25b-kube-api-access-fpl4f\") pod \"maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8\" (UID: \"ecb0e0c5-d2da-4624-a538-c872abaed25b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"
Oct 30 19:56:45.784043 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.783999    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:56:45.824266 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.824217    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9" event={"ID":"622c7980-cb92-4c47-ba6a-f333e5d6256e","Type":"ContainerStarted","Data":"60e00ecf0b8a265bf0ef904a2f8a00846f964be95fe3e09b8f10466a777e7cad"}
Oct 30 19:56:45.824742 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.824295    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9" podUID="622c7980-cb92-4c47-ba6a-f333e5d6256e" containerName="istio-proxy" containerID="cri-o://60e00ecf0b8a265bf0ef904a2f8a00846f964be95fe3e09b8f10466a777e7cad" gracePeriod=30
Oct 30 19:56:45.938815 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:45.938775    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"
Oct 30 19:56:46.239203 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:56:46.239173    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb0e0c5_d2da_4624_a538_c872abaed25b.slice/crio-886c1737ee11335447821a8219b384c047132a656cbc63668ad960567fdde799 WatchSource:0}: Error finding container 886c1737ee11335447821a8219b384c047132a656cbc63668ad960567fdde799: Status 404 returned error can't find the container with id 886c1737ee11335447821a8219b384c047132a656cbc63668ad960567fdde799
Oct 30 19:56:46.240540 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:46.240517    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"]
Oct 30 19:56:46.282509 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:46.282434    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a78c503-c362-4f0d-982a-9894bf6650dd" path="/var/lib/kubelet/pods/0a78c503-c362-4f0d-982a-9894bf6650dd/volumes"
Oct 30 19:56:46.836204 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:46.836163    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8" event={"ID":"ecb0e0c5-d2da-4624-a538-c872abaed25b","Type":"ContainerStarted","Data":"06366f3a1904b84da24a7ed55755962f04796df16bab68bd7a93296fe3fee2d4"}
Oct 30 19:56:46.836614 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:46.836229    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8" event={"ID":"ecb0e0c5-d2da-4624-a538-c872abaed25b","Type":"ContainerStarted","Data":"886c1737ee11335447821a8219b384c047132a656cbc63668ad960567fdde799"}
Oct 30 19:56:46.859745 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:46.859690    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8" podStartSLOduration=1.859672759 podStartE2EDuration="1.859672759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:46.855967856 +0000 UTC m=+1606.748582179" watchObservedRunningTime="2026-10-30 19:56:46.859672759 +0000 UTC m=+1606.752287085"
Oct 30 19:56:47.040391 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:47.040303    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"]
Oct 30 19:56:47.047337 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:47.047303    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2877b35f-7vcwg"]
Oct 30 19:56:47.847779 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:47.847736    4823 generic.go:356] "Generic (PLEG): container finished" podID="ecb0e0c5-d2da-4624-a538-c872abaed25b" containerID="06366f3a1904b84da24a7ed55755962f04796df16bab68bd7a93296fe3fee2d4" exitCode=0
Oct 30 19:56:47.848231 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:47.847794    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8" event={"ID":"ecb0e0c5-d2da-4624-a538-c872abaed25b","Type":"ContainerDied","Data":"06366f3a1904b84da24a7ed55755962f04796df16bab68bd7a93296fe3fee2d4"}
Oct 30 19:56:48.091885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:48.088274    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"]
Oct 30 19:56:48.094383 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:48.094338    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-450f5839-7vkhj"]
Oct 30 19:56:48.281067 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:48.281031    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc4c0f22-4317-4c34-ada6-2f40f7140a10" path="/var/lib/kubelet/pods/dc4c0f22-4317-4c34-ada6-2f40f7140a10/volumes"
Oct 30 19:56:48.281526 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:48.281511    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f4e928-2f39-4273-b567-e21b841e7ce5" path="/var/lib/kubelet/pods/e9f4e928-2f39-4273-b567-e21b841e7ce5/volumes"
Oct 30 19:56:49.132133 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:49.132102    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"
Oct 30 19:56:49.191505 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:49.191401    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb0e0c5-d2da-4624-a538-c872abaed25b-kube-api-access-fpl4f\" (UniqueName: \"kubernetes.io/projected/ecb0e0c5-d2da-4624-a538-c872abaed25b-kube-api-access-fpl4f\") pod \"ecb0e0c5-d2da-4624-a538-c872abaed25b\" (UID: \"ecb0e0c5-d2da-4624-a538-c872abaed25b\") "
Oct 30 19:56:49.191505 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:49.191458    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecb0e0c5-d2da-4624-a538-c872abaed25b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb0e0c5-d2da-4624-a538-c872abaed25b-maas-api-xks-ca\") pod \"ecb0e0c5-d2da-4624-a538-c872abaed25b\" (UID: \"ecb0e0c5-d2da-4624-a538-c872abaed25b\") "
Oct 30 19:56:49.191753 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:49.191680    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecb0e0c5-d2da-4624-a538-c872abaed25b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb0e0c5-d2da-4624-a538-c872abaed25b-maas-api-openshift-ca\") pod \"ecb0e0c5-d2da-4624-a538-c872abaed25b\" (UID: \"ecb0e0c5-d2da-4624-a538-c872abaed25b\") "
Oct 30 19:56:49.192203 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:49.192172    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb0e0c5-d2da-4624-a538-c872abaed25b-maas-api-openshift-ca" pod "ecb0e0c5-d2da-4624-a538-c872abaed25b" (UID: "ecb0e0c5-d2da-4624-a538-c872abaed25b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:49.192631 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:49.192610    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb0e0c5-d2da-4624-a538-c872abaed25b-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:49.194352 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:49.194327    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb0e0c5-d2da-4624-a538-c872abaed25b-maas-api-xks-ca" pod "ecb0e0c5-d2da-4624-a538-c872abaed25b" (UID: "ecb0e0c5-d2da-4624-a538-c872abaed25b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:49.194462 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:49.194395    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb0e0c5-d2da-4624-a538-c872abaed25b-kube-api-access-fpl4f" pod "ecb0e0c5-d2da-4624-a538-c872abaed25b" (UID: "ecb0e0c5-d2da-4624-a538-c872abaed25b"). InnerVolumeSpecName "kube-api-access-fpl4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:49.294342 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:49.294298    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpl4f\" (UniqueName: \"kubernetes.io/projected/ecb0e0c5-d2da-4624-a538-c872abaed25b-kube-api-access-fpl4f\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:49.294342 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:49.294334    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb0e0c5-d2da-4624-a538-c872abaed25b-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:49.939328 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:49.939302    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"
Oct 30 19:56:49.939567 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:49.939331    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8" event={"ID":"ecb0e0c5-d2da-4624-a538-c872abaed25b","Type":"ContainerDied","Data":"886c1737ee11335447821a8219b384c047132a656cbc63668ad960567fdde799"}
Oct 30 19:56:49.939567 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:49.939381    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="886c1737ee11335447821a8219b384c047132a656cbc63668ad960567fdde799"
Oct 30 19:56:50.955439 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:50.955405    4823 generic.go:356] "Generic (PLEG): container finished" podID="622c7980-cb92-4c47-ba6a-f333e5d6256e" containerID="60e00ecf0b8a265bf0ef904a2f8a00846f964be95fe3e09b8f10466a777e7cad" exitCode=0
Oct 30 19:56:50.955911 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:50.955475    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9" event={"ID":"622c7980-cb92-4c47-ba6a-f333e5d6256e","Type":"ContainerDied","Data":"60e00ecf0b8a265bf0ef904a2f8a00846f964be95fe3e09b8f10466a777e7cad"}
Oct 30 19:56:50.955911 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:50.955526    4823 scope.go:122] "RemoveContainer" containerID="bc999ee4a27e237ab48f4ae52ca6746dc9659fc769d227b354c062669f573e0e"
Oct 30 19:56:51.142125 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.142095    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:56:51.253938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.253855    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-data\") pod \"622c7980-cb92-4c47-ba6a-f333e5d6256e\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") "
Oct 30 19:56:51.254117 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.253955    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-credential-socket\") pod \"622c7980-cb92-4c47-ba6a-f333e5d6256e\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") "
Oct 30 19:56:51.254117 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.254073    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-workload-certs\") pod \"622c7980-cb92-4c47-ba6a-f333e5d6256e\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") "
Oct 30 19:56:51.254215 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.254127    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-envoy\") pod \"622c7980-cb92-4c47-ba6a-f333e5d6256e\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") "
Oct 30 19:56:51.254215 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.254147    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-data" pod "622c7980-cb92-4c47-ba6a-f333e5d6256e" (UID: "622c7980-cb92-4c47-ba6a-f333e5d6256e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:51.254300 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.254159    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/622c7980-cb92-4c47-ba6a-f333e5d6256e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/622c7980-cb92-4c47-ba6a-f333e5d6256e-istiod-ca-cert\") pod \"622c7980-cb92-4c47-ba6a-f333e5d6256e\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") "
Oct 30 19:56:51.254300 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.254279    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-credential-socket" pod "622c7980-cb92-4c47-ba6a-f333e5d6256e" (UID: "622c7980-cb92-4c47-ba6a-f333e5d6256e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:51.254415 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.254315    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-workload-socket\") pod \"622c7980-cb92-4c47-ba6a-f333e5d6256e\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") "
Oct 30 19:56:51.254415 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.254387    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-workload-certs" pod "622c7980-cb92-4c47-ba6a-f333e5d6256e" (UID: "622c7980-cb92-4c47-ba6a-f333e5d6256e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:51.254521 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.254506    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-workload-socket" pod "622c7980-cb92-4c47-ba6a-f333e5d6256e" (UID: "622c7980-cb92-4c47-ba6a-f333e5d6256e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:51.254583 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.254526    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-token\" (UniqueName: \"kubernetes.io/projected/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-token\") pod \"622c7980-cb92-4c47-ba6a-f333e5d6256e\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") "
Oct 30 19:56:51.254583 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.254568    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/622c7980-cb92-4c47-ba6a-f333e5d6256e-kube-api-access-f2xw6\" (UniqueName: \"kubernetes.io/projected/622c7980-cb92-4c47-ba6a-f333e5d6256e-kube-api-access-f2xw6\") pod \"622c7980-cb92-4c47-ba6a-f333e5d6256e\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") "
Oct 30 19:56:51.254678 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.254605    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/622c7980-cb92-4c47-ba6a-f333e5d6256e-istiod-ca-cert" pod "622c7980-cb92-4c47-ba6a-f333e5d6256e" (UID: "622c7980-cb92-4c47-ba6a-f333e5d6256e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:51.254678 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.254642    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-podinfo\") pod \"622c7980-cb92-4c47-ba6a-f333e5d6256e\" (UID: \"622c7980-cb92-4c47-ba6a-f333e5d6256e\") "
Oct 30 19:56:51.255373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.255349    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:51.255373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.255368    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/622c7980-cb92-4c47-ba6a-f333e5d6256e-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:51.255373 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.255379    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:51.255612 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.255388    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:51.255612 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.255397    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:51.256702 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.256673    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/622c7980-cb92-4c47-ba6a-f333e5d6256e-kube-api-access-f2xw6" pod "622c7980-cb92-4c47-ba6a-f333e5d6256e" (UID: "622c7980-cb92-4c47-ba6a-f333e5d6256e"). InnerVolumeSpecName "kube-api-access-f2xw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:51.256809 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.256781    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-podinfo" pod "622c7980-cb92-4c47-ba6a-f333e5d6256e" (UID: "622c7980-cb92-4c47-ba6a-f333e5d6256e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:56:51.256923 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.256907    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-token" pod "622c7980-cb92-4c47-ba6a-f333e5d6256e" (UID: "622c7980-cb92-4c47-ba6a-f333e5d6256e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:51.256999 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.256981    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-envoy" pod "622c7980-cb92-4c47-ba6a-f333e5d6256e" (UID: "622c7980-cb92-4c47-ba6a-f333e5d6256e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:51.357725 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.357611    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:51.357725 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.357654    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2xw6\" (UniqueName: \"kubernetes.io/projected/622c7980-cb92-4c47-ba6a-f333e5d6256e-kube-api-access-f2xw6\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:51.357725 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.357671    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:51.357725 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.357688    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/622c7980-cb92-4c47-ba6a-f333e5d6256e-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:51.966854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.966809    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9" event={"ID":"622c7980-cb92-4c47-ba6a-f333e5d6256e","Type":"ContainerDied","Data":"0f550f5e8e3f57b571c22d481ed415dc44fd46245a46e317cee58513d99fc319"}
Oct 30 19:56:51.966854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.966849    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"
Oct 30 19:56:51.967425 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:51.966938    4823 scope.go:122] "RemoveContainer" containerID="60e00ecf0b8a265bf0ef904a2f8a00846f964be95fe3e09b8f10466a777e7cad"
Oct 30 19:56:52.011166 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:52.011117    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"]
Oct 30 19:56:52.021485 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:52.021450    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b69bf89d-gw-openshift-default-6d558db8fc-xntq9"]
Oct 30 19:56:52.285935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:52.285768    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="622c7980-cb92-4c47-ba6a-f333e5d6256e" path="/var/lib/kubelet/pods/622c7980-cb92-4c47-ba6a-f333e5d6256e/volumes"
Oct 30 19:56:55.037651 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:55.037610    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"]
Oct 30 19:56:55.044062 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:55.044024    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a2bcef1-n5rs5"]
Oct 30 19:56:56.068405 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:56.068371    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"]
Oct 30 19:56:56.075014 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:56.074982    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335206cd-wrfwz"]
Oct 30 19:56:56.281244 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:56.281207    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c7794b-a90d-46ca-9667-1729a711596e" path="/var/lib/kubelet/pods/27c7794b-a90d-46ca-9667-1729a711596e/volumes"
Oct 30 19:56:56.281890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:56.281843    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b647870-c146-4318-816a-ba78907e8b10" path="/var/lib/kubelet/pods/5b647870-c146-4318-816a-ba78907e8b10/volumes"
Oct 30 19:56:59.052093 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:59.052054    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"]
Oct 30 19:56:59.073327 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:59.073270    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e7be96c-xhbhf"]
Oct 30 19:56:59.156084 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:56:59.156040    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:57:00.037946 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:00.037907    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"]
Oct 30 19:57:00.069888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:00.055780    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db4410c9-6f97m"]
Oct 30 19:57:00.327990 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:00.327480    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99954dc3-5d88-47ec-9408-164decaf5451" path="/var/lib/kubelet/pods/99954dc3-5d88-47ec-9408-164decaf5451/volumes"
Oct 30 19:57:00.329234 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:00.328606    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7b9f48-eab6-4e33-ac21-66265f054d57" path="/var/lib/kubelet/pods/bc7b9f48-eab6-4e33-ac21-66265f054d57/volumes"
Oct 30 19:57:02.053453 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:02.052781    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"]
Oct 30 19:57:02.285890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:02.181936    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7f2a520-cs7h8"]
Oct 30 19:57:02.509032 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:57:02.508819    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:57:02.573894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:02.573829    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb0e0c5-d2da-4624-a538-c872abaed25b" path="/var/lib/kubelet/pods/ecb0e0c5-d2da-4624-a538-c872abaed25b/volumes"
Oct 30 19:57:02.580899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:02.578401    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:02.816938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:02.811711    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:02.816938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:02.813768    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:57:02.875082 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:02.875037    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:03.322888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:03.294056    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:03.513152 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:03.513099    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:03.581085 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:03.565977    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:03.584770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:03.584718    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:03.585854 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:03.585827    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:57:03.753887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:03.751606    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"]
Oct 30 19:57:03.753887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:03.751953    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8" podUID="4253cfc3-c0b6-415b-919d-1fd829ab9ae8" containerName="istio-proxy" containerID="cri-o://11ea93849dd73719e5982bc3e81b0efe6ec06b0ca55c43c18be99f27be5bec24" gracePeriod=30
Oct 30 19:57:03.904481 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:03.890706    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:03.904481 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:03.892745    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:57:03.918926 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:03.918879    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:03.926383 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:03.926348    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:57:03.978496 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:03.978435    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv" podUID="44597884-0bca-4006-a0cd-6bf9344678bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:04.059613 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:04.059568    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:04.060049 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:04.060025    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-2wrpc" podUID="c2b23237-c311-4128-93c5-d6d8a3a53612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:57:04.156952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:04.156835    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:57:04.227910 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:04.190008    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:57:04.341572 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:04.295015    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:04.516947 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:04.511723    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-4zvvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:57:04.516947 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:04.511792    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg" podUID="11656a86-294a-4cd1-a29c-6f8d02a39c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:04.568191 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:04.568130    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:04.597004 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:04.588941    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:04.696887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:04.696024    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:04.752889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:04.752719    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:04.962784 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:04.961964    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:05.034368 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.031883    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:05.034368 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.032750    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:57:05.099832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.083368    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:05.192889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.160947    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:05.219947 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.201824    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"]
Oct 30 19:57:05.233350 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.233309    4823 cpu_manager.go:414] "containerMap: removing container" podUID="622c7980-cb92-4c47-ba6a-f333e5d6256e" containerName="istio-proxy"
Oct 30 19:57:05.233595 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.233574    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="622c7980-cb92-4c47-ba6a-f333e5d6256e" containerName="istio-proxy"
Oct 30 19:57:05.237745 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.237718    4823 cpu_manager.go:414] "containerMap: removing container" podUID="ecb0e0c5-d2da-4624-a538-c872abaed25b" containerName="revoke-keys"
Oct 30 19:57:05.241936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.241900    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb0e0c5-d2da-4624-a538-c872abaed25b" containerName="revoke-keys"
Oct 30 19:57:05.242236 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.242216    4823 cpu_manager.go:414] "containerMap: removing container" podUID="622c7980-cb92-4c47-ba6a-f333e5d6256e" containerName="istio-proxy"
Oct 30 19:57:05.245929 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.245897    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="622c7980-cb92-4c47-ba6a-f333e5d6256e" containerName="istio-proxy"
Oct 30 19:57:05.255897 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.246439    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="622c7980-cb92-4c47-ba6a-f333e5d6256e" containerName="istio-proxy"
Oct 30 19:57:05.256203 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.256154    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb0e0c5-d2da-4624-a538-c872abaed25b" containerName="revoke-keys"
Oct 30 19:57:05.305026 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.304990    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:57:05.305354 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.305328    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:57:05.305602 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.305583    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:57:05.305769 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.305753    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:57:05.309510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.309478    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:57:05.316934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.310900    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.373971 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.373930    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"]
Oct 30 19:57:05.382190 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.382050    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:05.427888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.404069    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.427888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.404192    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsfcf\" (UniqueName: \"kubernetes.io/projected/c965af44-94c0-4f66-8905-7d86c8b1f546-kube-api-access-nsfcf\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.427888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.404281    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.427888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.404355    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.427888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.404466    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.427888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.404703    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.427888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.404779    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.427888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.404827    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c965af44-94c0-4f66-8905-7d86c8b1f546-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.427888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.404901    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.444551 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.439314    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-b0d13a-openshift-default-dockercfg-b2mpb\""
Oct 30 19:57:05.517916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.515621    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsfcf\" (UniqueName: \"kubernetes.io/projected/c965af44-94c0-4f66-8905-7d86c8b1f546-kube-api-access-nsfcf\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.517916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.515719    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.517916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.515772    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.517916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.515844    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.517916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.516032    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.517916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.516099    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.517916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.516147    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c965af44-94c0-4f66-8905-7d86c8b1f546-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.517916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.516200    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.517916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.516283    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.517916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.516924    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.517916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.517514    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.517916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.517769    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.544182 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.526200    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.544182 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.526199    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c965af44-94c0-4f66-8905-7d86c8b1f546-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:05.656895 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.640846    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:05.706850 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.706793    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:57:05.805945 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.660180    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:57:05.805945 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:05.796386    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:57:06.053822 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.047530    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:06.093028 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.078137    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:57:06.139072 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.128215    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:57:06.172042 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.171971    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:57:06.314774 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.310638    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:57:06.342905 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.325705    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:06.343725 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.334578    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:06.343913 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.343566    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:06.395426 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.395348    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:06.436895 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.435205    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsfcf\" (UniqueName: \"kubernetes.io/projected/c965af44-94c0-4f66-8905-7d86c8b1f546-kube-api-access-nsfcf\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:06.615972 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.541710    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:06.627899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.622392    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-9djkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:57:06.627899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.622458    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx" podUID="88deb5dd-bf6b-406b-9aa9-103319a550e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:06.627899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.596545    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:06.679351 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.679309    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:57:06.708032 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.707977    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:06.802897 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.799445    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:06.803552 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:06.803518    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:57:07.140899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:07.128360    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:57:07.193077 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:07.185211    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:07.193077 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:07.141318    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:57:07.193077 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:07.153498    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:57:07.281924 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:07.156942    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:07.350915 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:07.328331    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:07.894034 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:07.893975    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:57:07.963095 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:07.963037    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:57:08.159026 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.158927    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:08.268887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.198011    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:08.268887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.198429    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:57:08.268887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.216124    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:08.268887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.222313    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:08.268887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.264501    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:08.358972 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.357321    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:08.643122 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.642666    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:57:08.643122 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.642846    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:57:08.643122 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.642988    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:57:08.877290 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.875686    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"]
Oct 30 19:57:08.886189 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.886141    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:57:08.911885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.910056    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:08.928281 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.928252    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:57:08.958651 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.958622    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:57:08.966659 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.964393    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:08.966659 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:08.965046    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:09.027368 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.013748    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:57:09.046067 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.045091    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:57:09.046246 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.046119    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:57:09.050245 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.049494    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:57:09.121307 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.120764    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"]
Oct 30 19:57:09.137911 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.136617    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d12a9b3f-sjqff"]
Oct 30 19:57:09.448425 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.448334    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" event={"ID":"c965af44-94c0-4f66-8905-7d86c8b1f546","Type":"ContainerStarted","Data":"0d4eff7631177d4ce9d48c8b7eb6b76a1a16b4989f9fb844c877477a941380c9"}
Oct 30 19:57:09.448425 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.448394    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" event={"ID":"c965af44-94c0-4f66-8905-7d86c8b1f546","Type":"ContainerStarted","Data":"cb8f279463b5a29a9b923409a3272fb8ffb9c14a184658e774b223cb92742e5e"}
Oct 30 19:57:09.521576 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.521515    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" podStartSLOduration=5.521491792 podStartE2EDuration="5.521491792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:09.477271626 +0000 UTC m=+1629.369885954" watchObservedRunningTime="2026-10-30 19:57:09.521491792 +0000 UTC m=+1629.414106120"
Oct 30 19:57:09.547769 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.547732    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:09.601981 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.601943    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"]
Oct 30 19:57:09.766114 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.765974    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="622c7980-cb92-4c47-ba6a-f333e5d6256e" containerName="istio-proxy"
Oct 30 19:57:09.933210 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.933177    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"]
Oct 30 19:57:09.934342 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.933319    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"
Oct 30 19:57:09.939654 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.939624    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:57:09.939780 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.939734    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:57:09.939851 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.939785    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:57:09.939851 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:09.939796    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:57:10.024532 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:10.024449    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed2ee64-b400-40ac-bc9f-24ecb430b169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0706d071-l7m89\" (UID: \"bed2ee64-b400-40ac-bc9f-24ecb430b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"
Oct 30 19:57:10.024729 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:10.024707    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djl4s\" (UniqueName: \"kubernetes.io/projected/bed2ee64-b400-40ac-bc9f-24ecb430b169-kube-api-access-djl4s\") pod \"maas-api-revoke-sub-0706d071-l7m89\" (UID: \"bed2ee64-b400-40ac-bc9f-24ecb430b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"
Oct 30 19:57:10.024783 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:10.024770    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed2ee64-b400-40ac-bc9f-24ecb430b169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0706d071-l7m89\" (UID: \"bed2ee64-b400-40ac-bc9f-24ecb430b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"
Oct 30 19:57:10.034914 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:10.034887    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"]
Oct 30 19:57:10.040834 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:10.040806    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f54eed2-jwdqh"]
Oct 30 19:57:10.126704 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:10.126670    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed2ee64-b400-40ac-bc9f-24ecb430b169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0706d071-l7m89\" (UID: \"bed2ee64-b400-40ac-bc9f-24ecb430b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"
Oct 30 19:57:10.126988 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:10.126890    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djl4s\" (UniqueName: \"kubernetes.io/projected/bed2ee64-b400-40ac-bc9f-24ecb430b169-kube-api-access-djl4s\") pod \"maas-api-revoke-sub-0706d071-l7m89\" (UID: \"bed2ee64-b400-40ac-bc9f-24ecb430b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"
Oct 30 19:57:10.126988 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:10.126941    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed2ee64-b400-40ac-bc9f-24ecb430b169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0706d071-l7m89\" (UID: \"bed2ee64-b400-40ac-bc9f-24ecb430b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"
Oct 30 19:57:10.127449 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:10.127428    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed2ee64-b400-40ac-bc9f-24ecb430b169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0706d071-l7m89\" (UID: \"bed2ee64-b400-40ac-bc9f-24ecb430b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"
Oct 30 19:57:10.129537 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:10.129515    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed2ee64-b400-40ac-bc9f-24ecb430b169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0706d071-l7m89\" (UID: \"bed2ee64-b400-40ac-bc9f-24ecb430b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"
Oct 30 19:57:10.139830 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:10.139797    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djl4s\" (UniqueName: \"kubernetes.io/projected/bed2ee64-b400-40ac-bc9f-24ecb430b169-kube-api-access-djl4s\") pod \"maas-api-revoke-sub-0706d071-l7m89\" (UID: \"bed2ee64-b400-40ac-bc9f-24ecb430b169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"
Oct 30 19:57:10.303944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:10.301394    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"
Oct 30 19:57:10.313994 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:10.313963    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6" path="/var/lib/kubelet/pods/4ec1e317-b121-4b97-bb23-9ee5ab2b2eb6/volumes"
Oct 30 19:57:10.314747 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:10.314730    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76698bf7-4947-4e62-8bda-b4e2a17c44f5" path="/var/lib/kubelet/pods/76698bf7-4947-4e62-8bda-b4e2a17c44f5/volumes"
Oct 30 19:57:10.586944 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:10.584492    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:10.749616 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:10.749579    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"]
Oct 30 19:57:11.047172 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:11.047142    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"]
Oct 30 19:57:11.057217 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:11.057179    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-033834de-czh2l"]
Oct 30 19:57:11.542952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:11.541600    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89" event={"ID":"bed2ee64-b400-40ac-bc9f-24ecb430b169","Type":"ContainerStarted","Data":"01e305f5e77f7fef928d0a05d8ff0803bd8dbe8cc4f537d841054ef22c13c93a"}
Oct 30 19:57:11.542952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:11.541668    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89" event={"ID":"bed2ee64-b400-40ac-bc9f-24ecb430b169","Type":"ContainerStarted","Data":"67246fbe6b21753052c7b85f816c30386ac99d691bb6a6ebef754b2f7d6df8bf"}
Oct 30 19:57:11.542952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:11.542096    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:11.543230 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:11.542959    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:57:11.569680 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:11.569610    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89" podStartSLOduration=2.5695885990000003 podStartE2EDuration="2.569588599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:11.55992372 +0000 UTC m=+1631.452538045" watchObservedRunningTime="2026-10-30 19:57:11.569588599 +0000 UTC m=+1631.462202923"
Oct 30 19:57:12.082675 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:12.082635    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"]
Oct 30 19:57:12.091935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:12.091897    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a7ab5b-4phst"]
Oct 30 19:57:12.285386 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:12.285304    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1583ba0-2c91-4348-bfd0-83e2b43bac2c" path="/var/lib/kubelet/pods/a1583ba0-2c91-4348-bfd0-83e2b43bac2c/volumes"
Oct 30 19:57:12.286084 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:12.286063    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc739f79-1809-42c5-8660-816cfa106a70" path="/var/lib/kubelet/pods/cc739f79-1809-42c5-8660-816cfa106a70/volumes"
Oct 30 19:57:12.556326 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:12.556148    4823 generic.go:356] "Generic (PLEG): container finished" podID="bed2ee64-b400-40ac-bc9f-24ecb430b169" containerID="01e305f5e77f7fef928d0a05d8ff0803bd8dbe8cc4f537d841054ef22c13c93a" exitCode=0
Oct 30 19:57:12.556980 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:12.556573    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89" event={"ID":"bed2ee64-b400-40ac-bc9f-24ecb430b169","Type":"ContainerDied","Data":"01e305f5e77f7fef928d0a05d8ff0803bd8dbe8cc4f537d841054ef22c13c93a"}
Oct 30 19:57:13.822410 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:13.822327    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"
Oct 30 19:57:14.001885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:13.994769    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed2ee64-b400-40ac-bc9f-24ecb430b169-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed2ee64-b400-40ac-bc9f-24ecb430b169-maas-api-openshift-ca\") pod \"bed2ee64-b400-40ac-bc9f-24ecb430b169\" (UID: \"bed2ee64-b400-40ac-bc9f-24ecb430b169\") "
Oct 30 19:57:14.001885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:13.994918    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed2ee64-b400-40ac-bc9f-24ecb430b169-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed2ee64-b400-40ac-bc9f-24ecb430b169-maas-api-xks-ca\") pod \"bed2ee64-b400-40ac-bc9f-24ecb430b169\" (UID: \"bed2ee64-b400-40ac-bc9f-24ecb430b169\") "
Oct 30 19:57:14.001885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:13.994998    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed2ee64-b400-40ac-bc9f-24ecb430b169-kube-api-access-djl4s\" (UniqueName: \"kubernetes.io/projected/bed2ee64-b400-40ac-bc9f-24ecb430b169-kube-api-access-djl4s\") pod \"bed2ee64-b400-40ac-bc9f-24ecb430b169\" (UID: \"bed2ee64-b400-40ac-bc9f-24ecb430b169\") "
Oct 30 19:57:14.001885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:13.998803    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed2ee64-b400-40ac-bc9f-24ecb430b169-maas-api-openshift-ca" pod "bed2ee64-b400-40ac-bc9f-24ecb430b169" (UID: "bed2ee64-b400-40ac-bc9f-24ecb430b169"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:14.131982 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:14.110917    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed2ee64-b400-40ac-bc9f-24ecb430b169-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.131982 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:14.114605    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed2ee64-b400-40ac-bc9f-24ecb430b169-maas-api-xks-ca" pod "bed2ee64-b400-40ac-bc9f-24ecb430b169" (UID: "bed2ee64-b400-40ac-bc9f-24ecb430b169"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:57:14.131982 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:14.115143    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed2ee64-b400-40ac-bc9f-24ecb430b169-kube-api-access-djl4s" pod "bed2ee64-b400-40ac-bc9f-24ecb430b169" (UID: "bed2ee64-b400-40ac-bc9f-24ecb430b169"). InnerVolumeSpecName "kube-api-access-djl4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:14.153884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:14.151805    4823 patch_prober.go:29] interesting pod/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:57:14.153884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:14.151881    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8" podUID="4253cfc3-c0b6-415b-919d-1fd829ab9ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:14.228887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:14.220103    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed2ee64-b400-40ac-bc9f-24ecb430b169-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.228887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:14.220156    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djl4s\" (UniqueName: \"kubernetes.io/projected/bed2ee64-b400-40ac-bc9f-24ecb430b169-kube-api-access-djl4s\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.696884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:14.695843    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89" event={"ID":"bed2ee64-b400-40ac-bc9f-24ecb430b169","Type":"ContainerDied","Data":"67246fbe6b21753052c7b85f816c30386ac99d691bb6a6ebef754b2f7d6df8bf"}
Oct 30 19:57:14.696884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:14.695931    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67246fbe6b21753052c7b85f816c30386ac99d691bb6a6ebef754b2f7d6df8bf"
Oct 30 19:57:14.696884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:14.696007    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"
Oct 30 19:57:17.129792 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.128696    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": read tcp 10.132.0.2:37486->10.132.0.139:15021: read: connection reset by peer" start-of-body=
Oct 30 19:57:17.129792 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.128807    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95" podUID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": read tcp 10.132.0.2:37486->10.132.0.139:15021: read: connection reset by peer"
Oct 30 19:57:17.420796 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.420768    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:57:17.563986 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.563946    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-credential-socket\") pod \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") "
Oct 30 19:57:17.564173 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.564074    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-podinfo\") pod \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") "
Oct 30 19:57:17.564241 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.564211    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-envoy\") pod \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") "
Oct 30 19:57:17.564300 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.564253    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-workload-socket\") pod \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") "
Oct 30 19:57:17.564300 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.564282    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-token\" (UniqueName: \"kubernetes.io/projected/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-token\") pod \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") "
Oct 30 19:57:17.564414 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.564287    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-credential-socket" pod "4253cfc3-c0b6-415b-919d-1fd829ab9ae8" (UID: "4253cfc3-c0b6-415b-919d-1fd829ab9ae8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:17.564414 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.564358    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-workload-certs\") pod \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") "
Oct 30 19:57:17.564518 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.564431    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istiod-ca-cert\") pod \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") "
Oct 30 19:57:17.564955 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.564924    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istiod-ca-cert" pod "4253cfc3-c0b6-415b-919d-1fd829ab9ae8" (UID: "4253cfc3-c0b6-415b-919d-1fd829ab9ae8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:17.565084 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.565039    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-workload-socket" pod "4253cfc3-c0b6-415b-919d-1fd829ab9ae8" (UID: "4253cfc3-c0b6-415b-919d-1fd829ab9ae8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:17.565084 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.565060    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-data\") pod \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") "
Oct 30 19:57:17.565198 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.565119    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-kube-api-access-fmhhv\" (UniqueName: \"kubernetes.io/projected/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-kube-api-access-fmhhv\") pod \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\" (UID: \"4253cfc3-c0b6-415b-919d-1fd829ab9ae8\") "
Oct 30 19:57:17.565309 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.565282    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-data" pod "4253cfc3-c0b6-415b-919d-1fd829ab9ae8" (UID: "4253cfc3-c0b6-415b-919d-1fd829ab9ae8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:17.566300 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.566271    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:17.566429 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.566304    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:17.567336 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.566999    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:17.567336 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.567027    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:17.571609 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.566281    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-workload-certs" pod "4253cfc3-c0b6-415b-919d-1fd829ab9ae8" (UID: "4253cfc3-c0b6-415b-919d-1fd829ab9ae8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:17.571727 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.567164    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-podinfo" pod "4253cfc3-c0b6-415b-919d-1fd829ab9ae8" (UID: "4253cfc3-c0b6-415b-919d-1fd829ab9ae8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:57:17.571727 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.567275    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-envoy" pod "4253cfc3-c0b6-415b-919d-1fd829ab9ae8" (UID: "4253cfc3-c0b6-415b-919d-1fd829ab9ae8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:17.571727 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.567420    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-token" pod "4253cfc3-c0b6-415b-919d-1fd829ab9ae8" (UID: "4253cfc3-c0b6-415b-919d-1fd829ab9ae8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:17.571727 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.568642    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-kube-api-access-fmhhv" pod "4253cfc3-c0b6-415b-919d-1fd829ab9ae8" (UID: "4253cfc3-c0b6-415b-919d-1fd829ab9ae8"). InnerVolumeSpecName "kube-api-access-fmhhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:17.668328 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.668234    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:17.668328 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.668269    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:17.668328 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.668279    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:17.668328 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.668290    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:17.668328 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.668300    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmhhv\" (UniqueName: \"kubernetes.io/projected/4253cfc3-c0b6-415b-919d-1fd829ab9ae8-kube-api-access-fmhhv\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:17.731497 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.731464    4823 generic.go:356] "Generic (PLEG): container finished" podID="4253cfc3-c0b6-415b-919d-1fd829ab9ae8" containerID="11ea93849dd73719e5982bc3e81b0efe6ec06b0ca55c43c18be99f27be5bec24" exitCode=0
Oct 30 19:57:17.731672 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.731514    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"
Oct 30 19:57:17.731672 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.731557    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8" event={"ID":"4253cfc3-c0b6-415b-919d-1fd829ab9ae8","Type":"ContainerDied","Data":"11ea93849dd73719e5982bc3e81b0efe6ec06b0ca55c43c18be99f27be5bec24"}
Oct 30 19:57:17.731672 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.731650    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8" event={"ID":"4253cfc3-c0b6-415b-919d-1fd829ab9ae8","Type":"ContainerDied","Data":"ef49d622e12835f8b700cc3d67b2367274c06dd68c7293ddc90b86c94029e333"}
Oct 30 19:57:17.731850 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.731682    4823 scope.go:122] "RemoveContainer" containerID="11ea93849dd73719e5982bc3e81b0efe6ec06b0ca55c43c18be99f27be5bec24"
Oct 30 19:57:17.734126 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.734096    4823 generic.go:356] "Generic (PLEG): container finished" podID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" containerID="af24e8b669cd122de5bdd2c3880f7b06e4a30aefdfb0f753d9590bfb47ee3c42" exitCode=0
Oct 30 19:57:17.734249 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.734161    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95" event={"ID":"7862ef1c-a91c-40f6-abaf-d5eba65b67c5","Type":"ContainerDied","Data":"af24e8b669cd122de5bdd2c3880f7b06e4a30aefdfb0f753d9590bfb47ee3c42"}
Oct 30 19:57:17.734767 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.734749    4823 scope.go:122] "RemoveContainer" containerID="af24e8b669cd122de5bdd2c3880f7b06e4a30aefdfb0f753d9590bfb47ee3c42"
Oct 30 19:57:17.736673 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.736638    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:57:17.736783 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.736722    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:57:17.736783 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.736756    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:57:17.749942 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.749917    4823 scope.go:122] "RemoveContainer" containerID="11ea93849dd73719e5982bc3e81b0efe6ec06b0ca55c43c18be99f27be5bec24"
Oct 30 19:57:17.750289 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:57:17.750262    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ea93849dd73719e5982bc3e81b0efe6ec06b0ca55c43c18be99f27be5bec24\": container with ID starting with 11ea93849dd73719e5982bc3e81b0efe6ec06b0ca55c43c18be99f27be5bec24 not found: ID does not exist" containerID="11ea93849dd73719e5982bc3e81b0efe6ec06b0ca55c43c18be99f27be5bec24"
Oct 30 19:57:17.750357 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.750299    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ea93849dd73719e5982bc3e81b0efe6ec06b0ca55c43c18be99f27be5bec24"} err="failed to get container status \"11ea93849dd73719e5982bc3e81b0efe6ec06b0ca55c43c18be99f27be5bec24\": rpc error: code = NotFound desc = could not find container \"11ea93849dd73719e5982bc3e81b0efe6ec06b0ca55c43c18be99f27be5bec24\": container with ID starting with 11ea93849dd73719e5982bc3e81b0efe6ec06b0ca55c43c18be99f27be5bec24 not found: ID does not exist"
Oct 30 19:57:17.794233 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.794194    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"]
Oct 30 19:57:17.802964 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:17.802927    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e7f2a520-openshift-default-85947d7cfc-l68t8"]
Oct 30 19:57:18.283571 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.283535    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4253cfc3-c0b6-415b-919d-1fd829ab9ae8" path="/var/lib/kubelet/pods/4253cfc3-c0b6-415b-919d-1fd829ab9ae8/volumes"
Oct 30 19:57:18.456886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.452127    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"]
Oct 30 19:57:18.456886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.453351    4823 cpu_manager.go:414] "containerMap: removing container" podUID="4253cfc3-c0b6-415b-919d-1fd829ab9ae8" containerName="istio-proxy"
Oct 30 19:57:18.456886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.453374    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4253cfc3-c0b6-415b-919d-1fd829ab9ae8" containerName="istio-proxy"
Oct 30 19:57:18.456886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.453414    4823 cpu_manager.go:414] "containerMap: removing container" podUID="bed2ee64-b400-40ac-bc9f-24ecb430b169" containerName="revoke-keys"
Oct 30 19:57:18.456886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.453424    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed2ee64-b400-40ac-bc9f-24ecb430b169" containerName="revoke-keys"
Oct 30 19:57:18.456886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.453677    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed2ee64-b400-40ac-bc9f-24ecb430b169" containerName="revoke-keys"
Oct 30 19:57:18.456886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.453707    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="4253cfc3-c0b6-415b-919d-1fd829ab9ae8" containerName="istio-proxy"
Oct 30 19:57:18.469883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.459925    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.489623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.488688    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-workload-certs\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.489623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.488769    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-workload-socket\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.489623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.488828    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-data\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.489623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.488924    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqrw\" (UniqueName: \"kubernetes.io/projected/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-kube-api-access-lrqrw\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.489623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.488973    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-token\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.489623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.489138    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istiod-ca-cert\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.489623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.489233    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ae43d70e-openshift-default-dockercfg-gh5t7\""
Oct 30 19:57:18.489623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.489346    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-podinfo\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.489623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.489418    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-envoy\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.489623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.489494    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-credential-socket\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.500300 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.497615    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"]
Oct 30 19:57:18.591091 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.590983    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-workload-socket\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.591091 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.591078    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-data\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.591420 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.591169    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqrw\" (UniqueName: \"kubernetes.io/projected/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-kube-api-access-lrqrw\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.591420 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.591219    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-token\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.591420 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.591257    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istiod-ca-cert\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.591420 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.591408    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-podinfo\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.591636 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.591468    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-envoy\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.591636 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.591538    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-credential-socket\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.591636 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.591650    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-workload-certs\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.592222 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.592193    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-workload-certs\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.592478 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.592456    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-workload-socket\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.592723 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.592703    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-data\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:18.666125 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:18.663295    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-credential-socket\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:19.077180 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:19.074601    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istiod-ca-cert\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:19.077180 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:19.075264    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-podinfo\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:19.077180 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:19.075906    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqrw\" (UniqueName: \"kubernetes.io/projected/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-kube-api-access-lrqrw\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:19.077923 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:19.077528    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-envoy\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:19.083423 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:19.083387    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-token\") pod \"e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") " pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:19.107008 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:19.104293    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:19.107008 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:19.104559    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95" event={"ID":"7862ef1c-a91c-40f6-abaf-d5eba65b67c5","Type":"ContainerStarted","Data":"1e88f27d7c5878928f0ae468e4a054544adf45912e600799202b2d33c8248651"}
Oct 30 19:57:19.110428 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:19.110398    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:57:19.110608 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:19.110579    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:57:19.110689 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:19.110634    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95" podUID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:57:19.448912 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:57:19.448853    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa02d0d0_81a7_4a03_9114_6561a01c3ffb.slice/crio-1dda84c3d19cabc59314d3af6e4768a73ef32b3716b1d2d271bc2074dea2163a WatchSource:0}: Error finding container 1dda84c3d19cabc59314d3af6e4768a73ef32b3716b1d2d271bc2074dea2163a: Status 404 returned error can't find the container with id 1dda84c3d19cabc59314d3af6e4768a73ef32b3716b1d2d271bc2074dea2163a
Oct 30 19:57:19.452829 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:19.452778    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:57:19.478586 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:19.478538    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"]
Oct 30 19:57:19.483187 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:19.483047    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:57:19.483187 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:19.483138    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:57:20.111315 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:20.111281    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:57:20.111515 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:20.111345    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95" podUID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:57:20.116216 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:20.116177    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" event={"ID":"aa02d0d0-81a7-4a03-9114-6561a01c3ffb","Type":"ContainerStarted","Data":"05a5e55cf6ae8b74925442f2851a2269a207f30441962c7467a853302a5f188a"}
Oct 30 19:57:20.116359 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:20.116243    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" event={"ID":"aa02d0d0-81a7-4a03-9114-6561a01c3ffb","Type":"ContainerStarted","Data":"1dda84c3d19cabc59314d3af6e4768a73ef32b3716b1d2d271bc2074dea2163a"}
Oct 30 19:57:20.170885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:20.166072    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" podStartSLOduration=2.166039981 podStartE2EDuration="2.166039981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:20.156477112 +0000 UTC m=+1640.049091437" watchObservedRunningTime="2026-10-30 19:57:20.166039981 +0000 UTC m=+1640.058654342"
Oct 30 19:57:21.105215 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:21.105171    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:21.111098 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:21.111071    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:21.111241 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:21.111222    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:57:21.111311 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:21.111274    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95" podUID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:57:21.126037 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:21.125988    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:21.128296 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:21.128255    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:57:22.110346 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:22.110309    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:57:22.110790 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:22.110400    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95" podUID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:57:23.112502 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:23.112467    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:57:23.113182 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:23.113130    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95" podUID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:57:24.110520 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:24.110484    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:57:24.110814 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:24.110782    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95" podUID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:57:24.116748 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:24.116714    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"]
Oct 30 19:57:24.127638 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:24.127597    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-304ef3c6-ghclh"]
Oct 30 19:57:24.293455 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:24.293418    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af718ff0-4c4e-43a3-b801-6adf02ea1223" path="/var/lib/kubelet/pods/af718ff0-4c4e-43a3-b801-6adf02ea1223/volumes"
Oct 30 19:57:25.059022 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:25.058415    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"]
Oct 30 19:57:25.085097 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:25.084548    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31b9b376-dvknj"]
Oct 30 19:57:25.184037 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:25.184006    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:57:25.184731 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:25.184709    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:57:25.188793 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:25.188764    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 19:57:26.298798 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:26.298757    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e3b378b-2bd6-4895-87bd-e80f26d52c98" path="/var/lib/kubelet/pods/5e3b378b-2bd6-4895-87bd-e80f26d52c98/volumes"
Oct 30 19:57:32.017319 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:57:32.017274    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:57:38.036917 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:38.036740    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:57:38.061885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:38.059762    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:38.221891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:38.210018    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:38.286886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:38.285974    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:57:38.286886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:38.286406    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:57:38.286886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:38.286685    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 19:57:38.328075 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:38.327957    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:38.434314 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:38.413926    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-6p5n6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:57:38.434611 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:38.434576    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6p5n6" podUID="892f7495-a9e2-489d-92c5-12f35e1e65f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:38.809888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:38.808022    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:38.898077 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:38.898026    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:39.040018 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.039963    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:39.041318 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.041284    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:57:39.266916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.258011    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:39.266916 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.258716    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:57:39.314896 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.291025    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:57:39.402211 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.402174    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:57:39.402508 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.315276    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:39.410329 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.405850    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:57:39.521064 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.520955    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:57:39.593890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.561704    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:57:39.681890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.620884    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:57:39.706469 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.705085    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 30 19:57:39.869950 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.738076    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Oct 30 19:57:39.958633 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.808942    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:57:39.958633 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.928548    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:57:39.985960 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.931841    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:39.985960 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:39.962391    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:57:40.042084 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:40.040841    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:57:40.308989 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:40.308043    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:40.314782 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:40.314738    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:57:40.347800 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:40.334982    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:40.460710 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:40.460294    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:40.583938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:40.575058    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:57:40.630898 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:40.630444    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:57:40.716912 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:40.716849    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"]
Oct 30 19:57:40.717211 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:40.717170    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a73316f-ksrbs"]
Oct 30 19:57:40.733224 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:40.731835    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:57:40.736271 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:40.736216    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:57:40.796884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:40.786999    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:57:40.799617 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:40.799589    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:57:40.831349 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:40.831313    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:57:40.849787 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:40.849698    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:57:42.286596 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:42.286560    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42128e23-e4a6-43d2-a22a-446c337ab727" path="/var/lib/kubelet/pods/42128e23-e4a6-43d2-a22a-446c337ab727/volumes"
Oct 30 19:57:43.728391 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:43.728353    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"]
Oct 30 19:57:43.735066 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:43.735033    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:43.748293 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:43.748270    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-3d0a78-openshift-default-dockercfg-tzwwl\""
Oct 30 19:57:43.846257 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:43.844358    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-workload-certs\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:43.846671 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:43.846634    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-podinfo\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:43.846951 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:43.846915    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-data\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:43.853908 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:43.847382    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-envoy\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:43.854248 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:43.854205    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv9mj\" (UniqueName: \"kubernetes.io/projected/ee23fe73-0972-42d0-b734-f4c3748354f9-kube-api-access-gv9mj\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:43.856898 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:43.854590    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-workload-socket\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:43.857119 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:43.857088    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee23fe73-0972-42d0-b734-f4c3748354f9-istiod-ca-cert\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:43.857306 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:43.857241    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-token\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:43.869042 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:43.868998    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-credential-socket\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:43.922909 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:43.919107    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"]
Oct 30 19:57:44.065996 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.035050    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv9mj\" (UniqueName: \"kubernetes.io/projected/ee23fe73-0972-42d0-b734-f4c3748354f9-kube-api-access-gv9mj\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.097067 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.097015    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-workload-socket\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.217994 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.113370    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-workload-socket\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.234984 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.218306    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee23fe73-0972-42d0-b734-f4c3748354f9-istiod-ca-cert\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.234984 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.230201    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-token\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.234984 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.230276    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-credential-socket\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.234984 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.230524    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-workload-certs\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.234984 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.230760    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-podinfo\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.234984 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.230852    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-data\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.234984 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.230999    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-envoy\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.236138 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.236104    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-workload-certs\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.249970 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.247367    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-data\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.249970 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.247660    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-credential-socket\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.257579 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.257541    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee23fe73-0972-42d0-b734-f4c3748354f9-istiod-ca-cert\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.318353 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.318237    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv9mj\" (UniqueName: \"kubernetes.io/projected/ee23fe73-0972-42d0-b734-f4c3748354f9-kube-api-access-gv9mj\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.362887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.361936    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-token\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.371483 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.371444    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-envoy\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.406466 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.406413    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-podinfo\") pod \"e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") " pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.411192 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.410928    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:44.720563 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.720529    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"]
Oct 30 19:57:44.720848 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:57:44.720824    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee23fe73_0972_42d0_b734_f4c3748354f9.slice/crio-6440e6754345281c6f03af7d72a64fdaf9008354d696396924daca6ff4a339b2 WatchSource:0}: Error finding container 6440e6754345281c6f03af7d72a64fdaf9008354d696396924daca6ff4a339b2: Status 404 returned error can't find the container with id 6440e6754345281c6f03af7d72a64fdaf9008354d696396924daca6ff4a339b2
Oct 30 19:57:44.723988 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.723952    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:57:44.724117 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.724023    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:57:44.724117 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.724060    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:57:44.757116 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:44.757081    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg" event={"ID":"ee23fe73-0972-42d0-b734-f4c3748354f9","Type":"ContainerStarted","Data":"6440e6754345281c6f03af7d72a64fdaf9008354d696396924daca6ff4a339b2"}
Oct 30 19:57:45.771460 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:45.771401    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg" event={"ID":"ee23fe73-0972-42d0-b734-f4c3748354f9","Type":"ContainerStarted","Data":"2ab25446eb3edfaf9652e1e8ae0dd54962f42ce73fd1b25bd157f97198a30115"}
Oct 30 19:57:45.826213 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:45.826152    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg" podStartSLOduration=2.826136413 podStartE2EDuration="2.826136413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:45.819929024 +0000 UTC m=+1665.712543374" watchObservedRunningTime="2026-10-30 19:57:45.826136413 +0000 UTC m=+1665.718750737"
Oct 30 19:57:46.413964 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:46.413921    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:46.422637 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:46.422601    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:46.784778 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:46.784690    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:46.785846 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:46.785822    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:57:49.818793 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:49.818703    4823 generic.go:356] "Generic (PLEG): container finished" podID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" containerID="6d7549911361640349a52a5388f57952e449b8f0a495191422cd0c13a9c248d1" exitCode=0
Oct 30 19:57:49.819298 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:49.818781    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn" event={"ID":"cfe9f1d8-aef1-41cd-b586-c0c073d9b574","Type":"ContainerDied","Data":"6d7549911361640349a52a5388f57952e449b8f0a495191422cd0c13a9c248d1"}
Oct 30 19:57:49.819466 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:49.819443    4823 scope.go:122] "RemoveContainer" containerID="6d7549911361640349a52a5388f57952e449b8f0a495191422cd0c13a9c248d1"
Oct 30 19:57:49.821781 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:49.821743    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:57:49.821910 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:49.821839    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:57:49.821979 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:49.821910    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:57:50.959492 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:50.959347    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn" event={"ID":"cfe9f1d8-aef1-41cd-b586-c0c073d9b574","Type":"ContainerStarted","Data":"9368584fc56201e94bc1ddecc78d79adc0425b33e08f8ef0b2b55da7e7afed66"}
Oct 30 19:57:51.090652 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:51.090614    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:57:51.091680 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:51.091654    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:57:51.091793 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:51.091708    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn" podUID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 19:57:52.096730 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:52.095155    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:57:52.096730 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:52.095221    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn" podUID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 19:57:53.095856 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:53.095804    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:57:53.096135 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:53.095929    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn" podUID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 19:57:54.090054 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:54.090016    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:57:54.090558 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:54.090528    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn" podUID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 19:57:54.688430 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:54.688390    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:57:55.090285 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:55.090197    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:57:55.090285 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:55.090255    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn" podUID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 19:57:56.092989 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:56.091175    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:57:56.092989 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:56.091254    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn" podUID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 19:57:57.165919 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:57.163334    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:57:57.165919 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:57:57.165009    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 19:58:00.773740 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:00.771886    4823 scope.go:122] "RemoveContainer" containerID="11267ff2c0418c2985e64e73944274ba0e802a28b9883068950834a1a9cbca37"
Oct 30 19:58:00.839181 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:00.839152    4823 scope.go:122] "RemoveContainer" containerID="8b89c56a018fa879ac72b69e436da8549685d90fc729ff837e5432798352cc51"
Oct 30 19:58:00.861238 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:00.861215    4823 scope.go:122] "RemoveContainer" containerID="47a66079d46d0d1cd904b752a7a0fa1204875f327f15df03ed025a29d858b7af"
Oct 30 19:58:00.878202 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:00.878084    4823 scope.go:122] "RemoveContainer" containerID="27f79e82c465856bafd07f5418bfc4720335b79fa9dc0a862077462991c17d99"
Oct 30 19:58:00.893346 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:00.893321    4823 scope.go:122] "RemoveContainer" containerID="53362280dd5a401fc0ce93aad0a804283e214a038d65cff9187e93ebe15a1417"
Oct 30 19:58:00.905569 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:00.905544    4823 scope.go:122] "RemoveContainer" containerID="f84ef62429fd5defca0d215c407ee44efd8fd4c3e0471e9e433a8c0cb0523742"
Oct 30 19:58:00.917234 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:00.917211    4823 scope.go:122] "RemoveContainer" containerID="1fac3582036d1604b9824547cd3e41a7e20e0bdf49914824a31a8c99d3302b1c"
Oct 30 19:58:00.929565 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:00.929545    4823 scope.go:122] "RemoveContainer" containerID="742c89f9b096c3bd610fc3ad60bf0ae4ea635715444b3d42a0edff05adba3230"
Oct 30 19:58:02.065260 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:58:02.065218    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:58:04.732378 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:04.732339    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:58:05.058206 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:05.058101    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"]
Oct 30 19:58:05.121689 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:05.121641    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d71580b-f8kwz"]
Oct 30 19:58:06.348204 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:06.348106    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6579b770-b19c-47c3-8a5b-60d24481aec9" path="/var/lib/kubelet/pods/6579b770-b19c-47c3-8a5b-60d24481aec9/volumes"
Oct 30 19:58:12.702258 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.702207    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"]
Oct 30 19:58:12.714495 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.713857    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.734033 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.733993    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"]
Oct 30 19:58:12.735187 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.735163    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-274922-openshift-default-dockercfg-ltdf4\""
Oct 30 19:58:12.870489 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.870444    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5bxb\" (UniqueName: \"kubernetes.io/projected/326b0731-4c61-494d-9de4-0ad913784cf1-kube-api-access-q5bxb\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.870677 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.870545    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326b0731-4c61-494d-9de4-0ad913784cf1-istio-token\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.870725 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.870708    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326b0731-4c61-494d-9de4-0ad913784cf1-istiod-ca-cert\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.870768 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.870743    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-workload-certs\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.870911 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.870883    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-istio-envoy\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.870989 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.870974    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326b0731-4c61-494d-9de4-0ad913784cf1-istio-podinfo\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.871050 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.871037    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-istio-data\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.871178 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.871164    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-credential-socket\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.871234 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.871219    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-workload-socket\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.973448 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.973343    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-credential-socket\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.973724 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.973544    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-workload-socket\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.973724 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.973611    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5bxb\" (UniqueName: \"kubernetes.io/projected/326b0731-4c61-494d-9de4-0ad913784cf1-kube-api-access-q5bxb\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.973724 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.973675    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326b0731-4c61-494d-9de4-0ad913784cf1-istio-token\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.973935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.973846    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326b0731-4c61-494d-9de4-0ad913784cf1-istiod-ca-cert\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.973935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.973905    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-workload-certs\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.974046 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.973953    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-istio-envoy\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.974046 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.974021    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326b0731-4c61-494d-9de4-0ad913784cf1-istio-podinfo\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.974182 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.974110    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-istio-data\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.975413 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.974655    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-workload-socket\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.975413 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.974819    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-istio-data\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:12.975413 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:12.974990    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-credential-socket\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:13.010642 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:13.007953    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326b0731-4c61-494d-9de4-0ad913784cf1-istiod-ca-cert\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:13.029541 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:13.011120    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-istio-envoy\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:13.029541 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:13.021735    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326b0731-4c61-494d-9de4-0ad913784cf1-istio-podinfo\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:13.066978 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:13.038104    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-workload-certs\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:13.067285 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:13.048851    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326b0731-4c61-494d-9de4-0ad913784cf1-istio-token\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:13.085887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:13.074736    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5bxb\" (UniqueName: \"kubernetes.io/projected/326b0731-4c61-494d-9de4-0ad913784cf1-kube-api-access-q5bxb\") pod \"e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") " pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:13.355936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:13.349964    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:13.736427 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:13.736393    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"]
Oct 30 19:58:13.745285 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:13.745242    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:13.745444 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:13.745335    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:13.745444 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:13.745380    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:14.536524 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:14.536477    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj" event={"ID":"326b0731-4c61-494d-9de4-0ad913784cf1","Type":"ContainerStarted","Data":"66f4a5087cb706b96c414e45c1189fb615fdd35058c63480c8d1b9a669e3350c"}
Oct 30 19:58:14.536748 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:14.536549    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj" event={"ID":"326b0731-4c61-494d-9de4-0ad913784cf1","Type":"ContainerStarted","Data":"9acadb7bb8cd99c137ce05e6c2e4d560ab2321bc67686888c9f48c81ba5e7f54"}
Oct 30 19:58:14.571199 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:14.571123    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj" podStartSLOduration=2.571101335 podStartE2EDuration="2.571101335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:14.560950459 +0000 UTC m=+1694.453564815" watchObservedRunningTime="2026-10-30 19:58:14.571101335 +0000 UTC m=+1694.463715660"
Oct 30 19:58:15.164826 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:15.164682    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"]
Oct 30 19:58:15.176856 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:15.176799    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c2592d-7kbvf"]
Oct 30 19:58:15.351019 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:15.350984    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:15.359799 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:15.359761    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:15.549926 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:15.549831    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:15.555183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:15.555144    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:58:16.283115 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:16.283081    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b991fb45-8e0f-4fba-93e1-cf0086a825fe" path="/var/lib/kubelet/pods/b991fb45-8e0f-4fba-93e1-cf0086a825fe/volumes"
Oct 30 19:58:16.939894 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:16.935558    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"]
Oct 30 19:58:17.213425 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.213345    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"]
Oct 30 19:58:17.213583 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.213467    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.218371 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.218336    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-eaa5cd-openshift-default-dockercfg-999c4\""
Oct 30 19:58:17.380889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.340997    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-workload-certs\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.380889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.341052    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-podinfo\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.380889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.341197    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-envoy\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.380889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.341345    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d703686a-ccb8-4f4a-8eac-15ce8d774149-istiod-ca-cert\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.380889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.341703    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-workload-socket\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.380889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.341746    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5p6h\" (UniqueName: \"kubernetes.io/projected/d703686a-ccb8-4f4a-8eac-15ce8d774149-kube-api-access-s5p6h\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.380889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.341815    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-data\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.380889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.341842    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-credential-socket\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.380889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.341890    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-token\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.455900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.452235    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-workload-certs\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.455900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.452295    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-podinfo\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.455900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.452412    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-envoy\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.455900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.452509    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d703686a-ccb8-4f4a-8eac-15ce8d774149-istiod-ca-cert\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.455900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.452730    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-workload-socket\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.455900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.452762    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5p6h\" (UniqueName: \"kubernetes.io/projected/d703686a-ccb8-4f4a-8eac-15ce8d774149-kube-api-access-s5p6h\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.455900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.452819    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-data\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.455900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.452846    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-credential-socket\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.455900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.452891    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-token\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.455900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.453845    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-workload-certs\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.455900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.454403    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-workload-socket\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.455900 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.455844    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d703686a-ccb8-4f4a-8eac-15ce8d774149-istiod-ca-cert\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.456698 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.456121    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-data\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.456698 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.456635    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-credential-socket\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.523055 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.522624    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-podinfo\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.533899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.532206    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-envoy\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.566883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.566347    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5p6h\" (UniqueName: \"kubernetes.io/projected/d703686a-ccb8-4f4a-8eac-15ce8d774149-kube-api-access-s5p6h\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.591887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.590348    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-token\") pod \"e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") " pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:17.851832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:17.851732    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:18.227194 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:18.227161    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"]
Oct 30 19:58:18.227919 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:58:18.227849    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd703686a_ccb8_4f4a_8eac_15ce8d774149.slice/crio-4642f056183949317e3dfeab028c0f7f91ef6c2c79273c5e2b2ca1958d01de3a WatchSource:0}: Error finding container 4642f056183949317e3dfeab028c0f7f91ef6c2c79273c5e2b2ca1958d01de3a: Status 404 returned error can't find the container with id 4642f056183949317e3dfeab028c0f7f91ef6c2c79273c5e2b2ca1958d01de3a
Oct 30 19:58:18.232165 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:18.232119    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:18.232279 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:18.232224    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:18.232338 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:18.232281    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:18.636780 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:18.636736    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x" event={"ID":"d703686a-ccb8-4f4a-8eac-15ce8d774149","Type":"ContainerStarted","Data":"fc6179186d5c5fe99146518ebf8c18f6151db8b7a762b9337b7b0f4457971e23"}
Oct 30 19:58:18.638853 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:18.636814    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x" event={"ID":"d703686a-ccb8-4f4a-8eac-15ce8d774149","Type":"ContainerStarted","Data":"4642f056183949317e3dfeab028c0f7f91ef6c2c79273c5e2b2ca1958d01de3a"}
Oct 30 19:58:18.692699 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:18.692638    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x" podStartSLOduration=2.692620034 podStartE2EDuration="2.692620034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:18.687526939 +0000 UTC m=+1698.580141261" watchObservedRunningTime="2026-10-30 19:58:18.692620034 +0000 UTC m=+1698.585234357"
Oct 30 19:58:18.852689 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:18.852652    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:19.359026 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.358986    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"]
Oct 30 19:58:19.363312 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.363287    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.411577 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.411545    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"]
Oct 30 19:58:19.488402 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.488364    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.488584 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.488540    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.488634 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.488589    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.488720 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.488694    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7gdh\" (UniqueName: \"kubernetes.io/projected/4379a4ca-7279-4f32-9b90-39baca931c17-kube-api-access-l7gdh\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.488842 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.488826    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.488890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.488876    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4379a4ca-7279-4f32-9b90-39baca931c17-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.488975 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.488961    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4379a4ca-7279-4f32-9b90-39baca931c17-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.489012 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.489000    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.489110 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.489096    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4379a4ca-7279-4f32-9b90-39baca931c17-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.590339 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.590298    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4379a4ca-7279-4f32-9b90-39baca931c17-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.590532 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.590372    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.590532 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.590503    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.590677 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.590561    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.590677 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.590614    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7gdh\" (UniqueName: \"kubernetes.io/projected/4379a4ca-7279-4f32-9b90-39baca931c17-kube-api-access-l7gdh\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.590820 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.590799    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.591029 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.591003    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.591148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.590937    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.591148 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.591086    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4379a4ca-7279-4f32-9b90-39baca931c17-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.591269 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.591201    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.591269 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.591221    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4379a4ca-7279-4f32-9b90-39baca931c17-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.591269 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.591253    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.591501 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.591288    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.591807 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.591785    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4379a4ca-7279-4f32-9b90-39baca931c17-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.593246 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.593225    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4379a4ca-7279-4f32-9b90-39baca931c17-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.593708 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.593687    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.629504 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.629435    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4379a4ca-7279-4f32-9b90-39baca931c17-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.646749 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.646714    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7gdh\" (UniqueName: \"kubernetes.io/projected/4379a4ca-7279-4f32-9b90-39baca931c17-kube-api-access-l7gdh\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.649995 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.649966    4823 generic.go:356] "Generic (PLEG): container finished" podID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerID="0d4eff7631177d4ce9d48c8b7eb6b76a1a16b4989f9fb844c877477a941380c9" exitCode=0
Oct 30 19:58:19.650127 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.650045    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" event={"ID":"c965af44-94c0-4f66-8905-7d86c8b1f546","Type":"ContainerDied","Data":"0d4eff7631177d4ce9d48c8b7eb6b76a1a16b4989f9fb844c877477a941380c9"}
Oct 30 19:58:19.650680 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.650658    4823 scope.go:122] "RemoveContainer" containerID="0d4eff7631177d4ce9d48c8b7eb6b76a1a16b4989f9fb844c877477a941380c9"
Oct 30 19:58:19.652370 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.652348    4823 generic.go:356] "Generic (PLEG): container finished" podID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerID="05a5e55cf6ae8b74925442f2851a2269a207f30441962c7467a853302a5f188a" exitCode=0
Oct 30 19:58:19.652476 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.652418    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" event={"ID":"aa02d0d0-81a7-4a03-9114-6561a01c3ffb","Type":"ContainerDied","Data":"05a5e55cf6ae8b74925442f2851a2269a207f30441962c7467a853302a5f188a"}
Oct 30 19:58:19.652802 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.652774    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:19.652922 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.652880    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:19.653000 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.652933    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:19.653286 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.653267    4823 scope.go:122] "RemoveContainer" containerID="05a5e55cf6ae8b74925442f2851a2269a207f30441962c7467a853302a5f188a"
Oct 30 19:58:19.654915 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.654880    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:19.655022 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.654971    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:19.655099 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.655018    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:19.684716 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.684690    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:19.881203 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:19.878205    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:20.035247 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:58:20.035204    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4379a4ca_7279_4f32_9b90_39baca931c17.slice/crio-108622826baacabfadcc48c16514cd4baa5156afe360b5b320cfa45cd0847dc5 WatchSource:0}: Error finding container 108622826baacabfadcc48c16514cd4baa5156afe360b5b320cfa45cd0847dc5: Status 404 returned error can't find the container with id 108622826baacabfadcc48c16514cd4baa5156afe360b5b320cfa45cd0847dc5
Oct 30 19:58:20.039642 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:20.039584    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:20.039768 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:20.039683    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:20.039768 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:20.039724    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:20.048170 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:20.044775    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"]
Oct 30 19:58:20.726410 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:20.724980    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw" event={"ID":"4379a4ca-7279-4f32-9b90-39baca931c17","Type":"ContainerStarted","Data":"eebe8c7f934783d682b69ecdb302391fef6f3d166baf46789aae1912bef4c26d"}
Oct 30 19:58:20.726410 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:20.725050    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw" event={"ID":"4379a4ca-7279-4f32-9b90-39baca931c17","Type":"ContainerStarted","Data":"108622826baacabfadcc48c16514cd4baa5156afe360b5b320cfa45cd0847dc5"}
Oct 30 19:58:20.733378 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:20.731366    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" event={"ID":"c965af44-94c0-4f66-8905-7d86c8b1f546","Type":"ContainerStarted","Data":"9b06bfcedddd5e9f159e43c5a14b2b0abfdf388dcd76852891dad656bafa45a3"}
Oct 30 19:58:20.739023 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:20.737978    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" event={"ID":"aa02d0d0-81a7-4a03-9114-6561a01c3ffb","Type":"ContainerStarted","Data":"878150ec1187b9e42f0aec4c8afff83f8da2668f29faa5b0a97778f9a41711c7"}
Oct 30 19:58:20.739023 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:20.738047    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:20.742277 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:20.742240    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:58:20.773908 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:20.773827    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw" podStartSLOduration=1.7738076459999998 podStartE2EDuration="1.773807646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:20.772347798 +0000 UTC m=+1700.664962123" watchObservedRunningTime="2026-10-30 19:58:20.773807646 +0000 UTC m=+1700.666421971"
Oct 30 19:58:21.110031 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:21.109994    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:58:21.110414 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:21.110384    4823 patch_prober.go:29] interesting pod/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:58:21.122889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:21.110600    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 19:58:21.126325 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:21.126143    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:58:21.691949 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:21.613028    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:58:21.691949 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:21.632966    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:58:21.691949 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:21.633027    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 19:58:21.736428 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:21.736389    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:21.906999 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:21.746228    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:58:21.907306 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:21.907274    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw" podUID="4379a4ca-7279-4f32-9b90-39baca931c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 19:58:22.107408 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:22.107366    4823 patch_prober.go:29] interesting pod/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:58:22.107684 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:22.107652    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 19:58:22.413941 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:22.412945    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:58:22.455104 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:22.455055    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:22.555334 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:22.555296    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:58:22.555630 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:22.555603    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 19:58:22.706942 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:22.706395    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:58:22.706942 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:22.706460    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw" podUID="4379a4ca-7279-4f32-9b90-39baca931c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 19:58:23.064009 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:23.062919    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"]
Oct 30 19:58:23.117429 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:23.116981    4823 patch_prober.go:29] interesting pod/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:58:23.117429 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:23.117047    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 19:58:23.162113 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:23.162070    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a68ad3c-jkpch"]
Oct 30 19:58:23.548050 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:23.547983    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:58:23.548497 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:23.548441    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 19:58:24.125963 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:24.110104    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:58:24.173299 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:24.173238    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw" podUID="4379a4ca-7279-4f32-9b90-39baca931c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 19:58:24.244142 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:24.239274    4823 patch_prober.go:29] interesting pod/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:58:24.244142 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:24.239956    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 19:58:24.742642 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:24.742597    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:58:24.743507 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:24.743468    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw" podUID="4379a4ca-7279-4f32-9b90-39baca931c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 19:58:24.793885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:24.793144    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:24.836888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:24.816465    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:58:24.836888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:24.816528    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 19:58:25.090936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:25.090169    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f21689d-3d9f-43f7-8927-bae70428d5e9" path="/var/lib/kubelet/pods/4f21689d-3d9f-43f7-8927-bae70428d5e9/volumes"
Oct 30 19:58:25.123887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:25.106338    4823 patch_prober.go:29] interesting pod/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:58:25.123887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:25.106401    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 19:58:25.549888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:25.549596    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:58:25.549888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:25.549655    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 19:58:25.691895 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:25.686592    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:58:25.691895 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:25.686672    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw" podUID="4379a4ca-7279-4f32-9b90-39baca931c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 19:58:26.107315 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:26.107272    4823 patch_prober.go:29] interesting pod/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:58:26.107656 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:26.107615    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 19:58:26.542548 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:26.542467    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:58:26.548242 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:26.548204    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:58:26.548458 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:26.548440    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 19:58:26.686340 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:26.686290    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:58:26.686968 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:26.686935    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw" podUID="4379a4ca-7279-4f32-9b90-39baca931c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 19:58:27.116889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:27.110032    4823 patch_prober.go:29] interesting pod/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:58:27.117228 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:27.117191    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 19:58:27.570391 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:27.570293    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:58:27.570661 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:27.570626    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 19:58:27.696120 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:27.696080    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:58:27.696610 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:27.696149    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw" podUID="4379a4ca-7279-4f32-9b90-39baca931c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 19:58:27.878899 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:27.858364    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:28.138943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:28.113280    4823 patch_prober.go:29] interesting pod/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:58:28.138943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:28.113346    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 19:58:28.170893 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:28.160061    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:58:28.203019 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:28.202962    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:28.766896 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:28.760966    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:58:28.829919 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:28.828931    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw" podUID="4379a4ca-7279-4f32-9b90-39baca931c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 19:58:28.850959 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:28.826062    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:28.851263 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:28.826967    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:58:28.884886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:28.826468    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:58:28.924164 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:28.907456    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:28.953281 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:28.948193    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:28.953281 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:28.951135    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:58:29.133887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.114499    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 30 19:58:29.133887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.114631    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 30 19:58:29.133887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.120337    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:29.133887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.132704    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:58:29.149186 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.149148    4823 patch_prober.go:29] interesting pod/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:58:29.149454 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.149421    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 19:58:29.169447 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.169398    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:29.192958 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.173657    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:58:29.337889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.332184    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:58:29.643994 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.624944    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:58:29.764447 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.764406    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:58:29.835039 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.834999    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:58:29.838714 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.838673    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw" podUID="4379a4ca-7279-4f32-9b90-39baca931c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 30 19:58:29.840679 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.840655    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 19:58:29.883886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.851782    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:29.883886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.852266    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:58:30.013937 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:29.994287    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:30.051241 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:30.051195    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:58:30.115019 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:30.114970    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 30 19:58:30.120911 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:30.119535    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:58:30.205816 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:30.205781    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"]
Oct 30 19:58:30.217050 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:30.217002    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cd1db51-mvltf"]
Oct 30 19:58:30.306098 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:30.305385    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01638ce8-8f1a-41ae-972c-503676f04eac" path="/var/lib/kubelet/pods/01638ce8-8f1a-41ae-972c-503676f04eac/volumes"
Oct 30 19:58:30.746898 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:30.745633    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:30.753888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:30.747533    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:30.761885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:30.754570    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 19:58:30.811042 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:30.811010    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:58:30.814058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:30.814016    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 19:58:32.017915 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:58:32.017876    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:58:40.045550 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:40.045517    4823 generic.go:356] "Generic (PLEG): container finished" podID="ee23fe73-0972-42d0-b734-f4c3748354f9" containerID="2ab25446eb3edfaf9652e1e8ae0dd54962f42ce73fd1b25bd157f97198a30115" exitCode=0
Oct 30 19:58:40.046112 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:40.045587    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg" event={"ID":"ee23fe73-0972-42d0-b734-f4c3748354f9","Type":"ContainerDied","Data":"2ab25446eb3edfaf9652e1e8ae0dd54962f42ce73fd1b25bd157f97198a30115"}
Oct 30 19:58:40.046182 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:40.046148    4823 scope.go:122] "RemoveContainer" containerID="2ab25446eb3edfaf9652e1e8ae0dd54962f42ce73fd1b25bd157f97198a30115"
Oct 30 19:58:40.048076 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:40.048049    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:40.048152 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:40.048118    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:40.048207 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:40.048154    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:41.072077 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:41.071998    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg" event={"ID":"ee23fe73-0972-42d0-b734-f4c3748354f9","Type":"ContainerStarted","Data":"b4765da50ae3065bada0594f2ed7ea89c389721c857a6ca03eecc337ee1caff7"}
Oct 30 19:58:41.412436 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:41.412398    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:58:41.412623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:41.412577    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 19:58:41.412668 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:41.412624    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg" podUID="ee23fe73-0972-42d0-b734-f4c3748354f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 19:58:42.412066 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:42.412031    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 19:58:42.412518 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:42.412090    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg" podUID="ee23fe73-0972-42d0-b734-f4c3748354f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 19:58:43.413357 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:43.413299    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 19:58:43.414035 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:43.413995    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg" podUID="ee23fe73-0972-42d0-b734-f4c3748354f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 19:58:44.343151 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:44.343106    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:58:44.412018 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:44.411983    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 19:58:44.412198 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:44.412053    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg" podUID="ee23fe73-0972-42d0-b734-f4c3748354f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 19:58:45.115340 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:45.115305    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"]
Oct 30 19:58:45.132386 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:45.130090    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-263ad6b7-2t7bh"]
Oct 30 19:58:45.411973 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:45.411843    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 19:58:45.411973 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:45.411962    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg" podUID="ee23fe73-0972-42d0-b734-f4c3748354f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 19:58:46.312724 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:46.312681    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f" path="/var/lib/kubelet/pods/2b9c3d3f-fe44-454a-a2d3-11c3d8618d8f/volumes"
Oct 30 19:58:46.412474 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:46.412429    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 19:58:46.412712 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:46.412535    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg" podUID="ee23fe73-0972-42d0-b734-f4c3748354f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 19:58:46.785141 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:46.785102    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:58:47.094623 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:47.094518    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"]
Oct 30 19:58:47.105332 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:47.105263    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c330f80-j6f85"]
Oct 30 19:58:47.417584 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:47.417550    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:58:47.418812 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:47.418781    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 19:58:48.281521 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:48.281484    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="400327da-db74-4cce-be08-c75a73263374" path="/var/lib/kubelet/pods/400327da-db74-4cce-be08-c75a73263374/volumes"
Oct 30 19:58:50.760035 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:50.760001    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"]
Oct 30 19:58:50.764330 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:50.764307    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:50.809353 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:50.809320    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-98a0d8a9-openshift-default-dockercfg-m2zhg\""
Oct 30 19:58:50.823468 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:50.823431    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"]
Oct 30 19:58:50.860787 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:50.860750    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-credential-socket\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:50.861002 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:50.860908    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6660e61-7ad3-45cd-873e-d4c0a1741317-istiod-ca-cert\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:50.861002 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:50.860952    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-envoy\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:50.861203 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:50.861181    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-data\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:50.861289 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:50.861271    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-workload-certs\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:50.861380 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:50.861368    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-podinfo\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:50.861572 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:50.861544    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxpzx\" (UniqueName: \"kubernetes.io/projected/f6660e61-7ad3-45cd-873e-d4c0a1741317-kube-api-access-dxpzx\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:50.861642 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:50.861609    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-token\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:50.861795 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:50.861776    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-workload-socket\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.015180 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.015088    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-token\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.015919 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.015890    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-workload-socket\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.016585 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.016567    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-credential-socket\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.017106 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.017088    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6660e61-7ad3-45cd-873e-d4c0a1741317-istiod-ca-cert\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.028084 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.028051    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-envoy\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.033105 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.033073    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-data\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.033796 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.033773    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-workload-certs\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.054491 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.054460    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-podinfo\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.055270 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.055248    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxpzx\" (UniqueName: \"kubernetes.io/projected/f6660e61-7ad3-45cd-873e-d4c0a1741317-kube-api-access-dxpzx\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.056119 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.033678    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6660e61-7ad3-45cd-873e-d4c0a1741317-istiod-ca-cert\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.056281 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.016993    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-credential-socket\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.056419 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.053815    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-workload-certs\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.056520 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.033610    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-data\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.056610 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.016437    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-workload-socket\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.092064 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.091997    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-podinfo\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.098884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.098661    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxpzx\" (UniqueName: \"kubernetes.io/projected/f6660e61-7ad3-45cd-873e-d4c0a1741317-kube-api-access-dxpzx\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.108645 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.107331    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-envoy\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.109398 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.109369    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-token\") pod \"e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") " pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.379902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.379852    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:51.711087 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:58:51.711052    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6660e61_7ad3_45cd_873e_d4c0a1741317.slice/crio-6f02dac36f813be1aa5f0a353dfdcfb48367f2bd927e77dede2f2e1810683885 WatchSource:0}: Error finding container 6f02dac36f813be1aa5f0a353dfdcfb48367f2bd927e77dede2f2e1810683885: Status 404 returned error can't find the container with id 6f02dac36f813be1aa5f0a353dfdcfb48367f2bd927e77dede2f2e1810683885
Oct 30 19:58:51.714625 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.714597    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"]
Oct 30 19:58:51.715547 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.715507    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:51.715639 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.715589    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:51.715639 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:51.715634    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:52.416201 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:52.416160    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx" event={"ID":"f6660e61-7ad3-45cd-873e-d4c0a1741317","Type":"ContainerStarted","Data":"ea9a871b7f3e8a98eac8e369bc9ddb8e7687c6acde96b1d79b741f830fa82a6e"}
Oct 30 19:58:52.416645 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:52.416222    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx" event={"ID":"f6660e61-7ad3-45cd-873e-d4c0a1741317","Type":"ContainerStarted","Data":"6f02dac36f813be1aa5f0a353dfdcfb48367f2bd927e77dede2f2e1810683885"}
Oct 30 19:58:52.441145 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:52.441089    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx" podStartSLOduration=2.44107287 podStartE2EDuration="2.44107287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:52.438810765 +0000 UTC m=+1732.331425088" watchObservedRunningTime="2026-10-30 19:58:52.44107287 +0000 UTC m=+1732.333687194"
Oct 30 19:58:53.380770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:53.380733    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:53.388095 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:53.388064    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:53.426593 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:53.426555    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:53.427588 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:53.427567    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 19:58:58.757084 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.757038    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"]
Oct 30 19:58:58.763015 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.762837    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.772645 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.772613    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ae2147-openshift-default-dockercfg-x9mn6\""
Oct 30 19:58:58.781405 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.781373    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"]
Oct 30 19:58:58.826945 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.826883    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdk56\" (UniqueName: \"kubernetes.io/projected/778e153e-48af-4716-9fb2-2095e02c7bc2-kube-api-access-cdk56\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.827113 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.827034    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.827238 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.827217    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.827277 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.827259    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.827388 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.827371    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.827433 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.827406    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.827598 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.827575    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.827638 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.827619    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778e153e-48af-4716-9fb2-2095e02c7bc2-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.827781 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.827764    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.931132 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.930239    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.931132 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.930298    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.931132 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.930391    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.931132 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.930418    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778e153e-48af-4716-9fb2-2095e02c7bc2-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.931132 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.930526    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.931132 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.930748    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdk56\" (UniqueName: \"kubernetes.io/projected/778e153e-48af-4716-9fb2-2095e02c7bc2-kube-api-access-cdk56\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.931132 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.930787    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.931132 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.930923    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.931132 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.930961    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.932542 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.932513    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.950966 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.933700    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778e153e-48af-4716-9fb2-2095e02c7bc2-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.952807 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.952774    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.953633 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.953605    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.954052 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.954029    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.976431 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.976386    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.983889 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.981396    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.984238 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.984210    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdk56\" (UniqueName: \"kubernetes.io/projected/778e153e-48af-4716-9fb2-2095e02c7bc2-kube-api-access-cdk56\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:58.984629 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:58.984596    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:59.103943 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:59.100208    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:58:59.654712 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:58:59.654681    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778e153e_48af_4716_9fb2_2095e02c7bc2.slice/crio-1a88b234f4e71ade191b65268336b577c4d3920ec8948c01a4dcb4f4fc92a9f8 WatchSource:0}: Error finding container 1a88b234f4e71ade191b65268336b577c4d3920ec8948c01a4dcb4f4fc92a9f8: Status 404 returned error can't find the container with id 1a88b234f4e71ade191b65268336b577c4d3920ec8948c01a4dcb4f4fc92a9f8
Oct 30 19:58:59.654822 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:59.654727    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"]
Oct 30 19:58:59.658084 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:59.658033    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:59.658204 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:59.658123    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:58:59.658204 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:58:59.658154    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:00.036284 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:00.036195    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"]
Oct 30 19:59:00.042709 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:00.042675    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"]
Oct 30 19:59:00.052012 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:00.051965    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4701206e-4vbjs"]
Oct 30 19:59:00.059418 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:00.059385    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29e7fc77-xj7cz"]
Oct 30 19:59:00.282774 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:00.282733    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48611f0a-b2a4-4499-a80e-5d2ef398981a" path="/var/lib/kubelet/pods/48611f0a-b2a4-4499-a80e-5d2ef398981a/volumes"
Oct 30 19:59:00.283448 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:00.283426    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbdb0935-22a7-45e8-a4a2-a7053e726b8b" path="/var/lib/kubelet/pods/dbdb0935-22a7-45e8-a4a2-a7053e726b8b/volumes"
Oct 30 19:59:00.564651 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:00.564607    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25" event={"ID":"778e153e-48af-4716-9fb2-2095e02c7bc2","Type":"ContainerStarted","Data":"a13408882b277f94c3e5ec6cc9f708b7f0d754940aa41676980403d80cfa3938"}
Oct 30 19:59:00.564841 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:00.564670    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25" event={"ID":"778e153e-48af-4716-9fb2-2095e02c7bc2","Type":"ContainerStarted","Data":"1a88b234f4e71ade191b65268336b577c4d3920ec8948c01a4dcb4f4fc92a9f8"}
Oct 30 19:59:00.588489 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:00.588431    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25" podStartSLOduration=2.588415435 podStartE2EDuration="2.588415435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:00.586832556 +0000 UTC m=+1740.479446880" watchObservedRunningTime="2026-10-30 19:59:00.588415435 +0000 UTC m=+1740.481029758"
Oct 30 19:59:01.029059 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:01.029025    4823 scope.go:122] "RemoveContainer" containerID="9054dfe8c7832ce80e14f85a963de58d9d1be519f9a891c5cc7efee59a2f89c2"
Oct 30 19:59:01.042101 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:01.042075    4823 scope.go:122] "RemoveContainer" containerID="22cf4bf4655675e073e60393369d68b7e3886af022df2f5bf572fc8fae84e6ed"
Oct 30 19:59:01.054585 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:01.054564    4823 scope.go:122] "RemoveContainer" containerID="e924d470dbab6907de681b941b09576a4071b6e6ec8092dfddb9df0000089ad6"
Oct 30 19:59:01.066668 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:01.066648    4823 scope.go:122] "RemoveContainer" containerID="ec2fcd056bed347ab93a4cc248f466b440dfca06a8d0e60d0eec97fe0ae4b5e4"
Oct 30 19:59:01.078594 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:01.078564    4823 scope.go:122] "RemoveContainer" containerID="e371d229d89f18cfd51e883321a39e606bf40b22c77df3532de60c015d82a074"
Oct 30 19:59:01.090357 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:01.090335    4823 scope.go:122] "RemoveContainer" containerID="7f1aece4f3ce712f60ad2a7f7b5f13bc6a0167e1c67e5553798c8ed875f3f1f1"
Oct 30 19:59:01.100770 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:01.100741    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:59:01.102640 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:01.102619    4823 scope.go:122] "RemoveContainer" containerID="0c58fede203767ba8bc9bdd8f109150df795dc63f1482605d1b9eb99911007a6"
Oct 30 19:59:01.108157 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:01.108130    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:59:01.706320 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:01.701558    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:59:01.721886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:01.720152    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 19:59:02.071319 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:59:02.071233    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:59:03.663886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:03.659813    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"]
Oct 30 19:59:03.788891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:03.783113    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"]
Oct 30 19:59:03.788891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:03.783249    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:03.800921 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:03.795612    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c4c072-openshift-default-dockercfg-fqrfv\""
Oct 30 19:59:03.900793 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:03.900749    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7r85\" (UniqueName: \"kubernetes.io/projected/c51ac612-1da2-4152-978a-78726d5ea888-kube-api-access-r7r85\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:03.910015 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:03.909970    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-workload-certs\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:03.953107 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:03.953001    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c51ac612-1da2-4152-978a-78726d5ea888-istio-token\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:03.978936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:03.953453    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-istio-data\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:03.979302 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:03.979266    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-workload-socket\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.012937 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:03.983132    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c51ac612-1da2-4152-978a-78726d5ea888-istio-podinfo\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.035893 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.013427    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c51ac612-1da2-4152-978a-78726d5ea888-istiod-ca-cert\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.061038 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.060989    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-credential-socket\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.061262 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.061241    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-istio-envoy\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.216947 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.212893    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-credential-socket\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.229464 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.223334    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-credential-socket\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.229464 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.226996    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-istio-envoy\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.229464 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.227219    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7r85\" (UniqueName: \"kubernetes.io/projected/c51ac612-1da2-4152-978a-78726d5ea888-kube-api-access-r7r85\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.229464 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.227299    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-workload-certs\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.229464 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.227455    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c51ac612-1da2-4152-978a-78726d5ea888-istio-token\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.229464 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.227611    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-istio-data\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.229464 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.227680    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-workload-socket\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.229464 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.227977    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c51ac612-1da2-4152-978a-78726d5ea888-istio-podinfo\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.235932 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.233680    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-workload-certs\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.235932 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.234320    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-istio-data\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.235932 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.234553    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-workload-socket\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.247956 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.240729    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c51ac612-1da2-4152-978a-78726d5ea888-istio-podinfo\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.248117 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.248090    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c51ac612-1da2-4152-978a-78726d5ea888-istiod-ca-cert\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.249220 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.248913    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c51ac612-1da2-4152-978a-78726d5ea888-istiod-ca-cert\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.253510 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.253475    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c51ac612-1da2-4152-978a-78726d5ea888-istio-token\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.254051 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.254005    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7r85\" (UniqueName: \"kubernetes.io/projected/c51ac612-1da2-4152-978a-78726d5ea888-kube-api-access-r7r85\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.258335 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.257542    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-istio-envoy\") pod \"e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") " pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.326565 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.326527    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:04.497947 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.495091    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"]
Oct 30 19:59:04.513777 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.509102    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"]
Oct 30 19:59:04.513777 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.509229    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.578205 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.578160    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4fgh\" (UniqueName: \"kubernetes.io/projected/44eb96c1-778e-433f-ae45-3a42beed5dac-kube-api-access-w4fgh\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.578205 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.578222    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.578488 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.578345    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.578488 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.578407    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.578612 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.578583    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.578698 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.578681    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.578755 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.578735    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eb96c1-778e-433f-ae45-3a42beed5dac-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.578965 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.578940    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.579073 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.579053    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.680451 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.680409    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.681353 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.680616    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eb96c1-778e-433f-ae45-3a42beed5dac-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.681353 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.680952    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.681353 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.681059    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.681554 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.681476    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.681679 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.681552    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.681679 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.681665    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4fgh\" (UniqueName: \"kubernetes.io/projected/44eb96c1-778e-433f-ae45-3a42beed5dac-kube-api-access-w4fgh\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.681791 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.681701    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.681791 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.681721    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eb96c1-778e-433f-ae45-3a42beed5dac-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.681791 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.681787    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.681986 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.681788    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.681986 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.681882    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.682078 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.682001    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.682078 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.682011    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.684683 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.684651    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.684789 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.684659    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.707644 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.707611    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.715043 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.715011    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4fgh\" (UniqueName: \"kubernetes.io/projected/44eb96c1-778e-433f-ae45-3a42beed5dac-kube-api-access-w4fgh\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:04.764714 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:59:04.764677    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51ac612_1da2_4152_978a_78726d5ea888.slice/crio-82fa69f4d7fa880bde3651042944cca0315ce3479108855f9768e8a8ffffb922 WatchSource:0}: Error finding container 82fa69f4d7fa880bde3651042944cca0315ce3479108855f9768e8a8ffffb922: Status 404 returned error can't find the container with id 82fa69f4d7fa880bde3651042944cca0315ce3479108855f9768e8a8ffffb922
Oct 30 19:59:04.765994 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.765964    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"]
Oct 30 19:59:04.768086 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.768049    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:04.768180 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.768118    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:04.768180 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.768147    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:04.854239 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:04.854201    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:05.706676 ip-10-0-138-61 kubenswrapper[4823]: W1030 19:59:05.706640    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44eb96c1_778e_433f_ae45_3a42beed5dac.slice/crio-b0246e5bc68241fbec8647a1ebf7213ce318c6e98c6aa0dab0cf2f0090cd9986 WatchSource:0}: Error finding container b0246e5bc68241fbec8647a1ebf7213ce318c6e98c6aa0dab0cf2f0090cd9986: Status 404 returned error can't find the container with id b0246e5bc68241fbec8647a1ebf7213ce318c6e98c6aa0dab0cf2f0090cd9986
Oct 30 19:59:05.707816 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:05.707785    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"]
Oct 30 19:59:05.710096 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:05.710064    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:05.710192 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:05.710136    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:05.710192 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:05.710174    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:05.788708 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:05.788671    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc" event={"ID":"44eb96c1-778e-433f-ae45-3a42beed5dac","Type":"ContainerStarted","Data":"b0246e5bc68241fbec8647a1ebf7213ce318c6e98c6aa0dab0cf2f0090cd9986"}
Oct 30 19:59:05.790962 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:05.790924    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5" event={"ID":"c51ac612-1da2-4152-978a-78726d5ea888","Type":"ContainerStarted","Data":"f08e2a7bdcb16b2064e9884c5ef38f7c4cc455ef3b3e02e890d2675e7664ed84"}
Oct 30 19:59:05.791106 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:05.790980    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5" event={"ID":"c51ac612-1da2-4152-978a-78726d5ea888","Type":"ContainerStarted","Data":"82fa69f4d7fa880bde3651042944cca0315ce3479108855f9768e8a8ffffb922"}
Oct 30 19:59:05.818353 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:05.818289    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5" podStartSLOduration=2.818266879 podStartE2EDuration="2.818266879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:05.815540357 +0000 UTC m=+1745.708154693" watchObservedRunningTime="2026-10-30 19:59:05.818266879 +0000 UTC m=+1745.710881207"
Oct 30 19:59:06.357409 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:06.356285    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:06.357409 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:06.356459    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:06.821153 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:06.821053    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc" event={"ID":"44eb96c1-778e-433f-ae45-3a42beed5dac","Type":"ContainerStarted","Data":"671e1f1c862980d941a1747b1d9975fcf00760e7b94e103e1af167f3e9a1ad96"}
Oct 30 19:59:06.821620 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:06.821594    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:06.822641 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:06.822615    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 19:59:06.849377 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:06.847310    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc" podStartSLOduration=2.847290489 podStartE2EDuration="2.847290489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:06.845708999 +0000 UTC m=+1746.738323322" watchObservedRunningTime="2026-10-30 19:59:06.847290489 +0000 UTC m=+1746.739904813"
Oct 30 19:59:06.880105 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:06.880047    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:06.892316 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:06.884313    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:59:06.892316 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:06.884375    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc" podUID="44eb96c1-778e-433f-ae45-3a42beed5dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:59:07.879901 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:07.879803    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:59:07.880409 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:07.879915    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc" podUID="44eb96c1-778e-433f-ae45-3a42beed5dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:59:08.021440 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.021398    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"]
Oct 30 19:59:08.026872 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.026834    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.046247 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.046220    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dd1b04-openshift-default-dockercfg-dhzsm\""
Oct 30 19:59:08.214218 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.193361    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-credential-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.214218 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.193456    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqkrq\" (UniqueName: \"kubernetes.io/projected/dfc171b1-8278-449b-aea3-e0271b0b525d-kube-api-access-mqkrq\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.214218 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.193520    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-data\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.214218 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.193623    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-podinfo\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.214218 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.193663    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-envoy\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.214218 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.193707    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-workload-certs\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.214218 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.193735    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-workload-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.214218 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.193796    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc171b1-8278-449b-aea3-e0271b0b525d-istiod-ca-cert\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.257910 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.244114    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-token\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.361907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.357440    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-credential-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.361907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.357552    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqkrq\" (UniqueName: \"kubernetes.io/projected/dfc171b1-8278-449b-aea3-e0271b0b525d-kube-api-access-mqkrq\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.361907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.357621    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-data\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.361907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.357728    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-podinfo\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.361907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.357769    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-envoy\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.361907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.357817    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-workload-certs\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.361907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.357846    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-workload-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.361907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.357957    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc171b1-8278-449b-aea3-e0271b0b525d-istiod-ca-cert\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.361907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.358122    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-token\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.361907 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.359708    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-credential-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.411918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.402851    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-workload-certs\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.411918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.403236    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-workload-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.411918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.403949    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc171b1-8278-449b-aea3-e0271b0b525d-istiod-ca-cert\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.411918 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.404582    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-data\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.620030 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.602833    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-podinfo\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.642156 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.624766    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"]
Oct 30 19:59:08.666358 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.666326    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-envoy\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.668932 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.668441    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-token\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.743352 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.739223    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqkrq\" (UniqueName: \"kubernetes.io/projected/dfc171b1-8278-449b-aea3-e0271b0b525d-kube-api-access-mqkrq\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:08.872279 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.860679    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:59:08.872279 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:08.860743    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc" podUID="44eb96c1-778e-433f-ae45-3a42beed5dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:59:09.010888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:09.008527    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:09.497845 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:09.497602    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"]
Oct 30 19:59:09.499283 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:09.499248    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:09.499407 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:09.499328    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:09.499407 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:09.499373    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:09.859896 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:09.859839    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:59:09.860158 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:09.860127    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc" podUID="44eb96c1-778e-433f-ae45-3a42beed5dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:59:09.927996 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:09.926325    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt" event={"ID":"dfc171b1-8278-449b-aea3-e0271b0b525d","Type":"ContainerStarted","Data":"a49165c07832a8da66130d1d6106d1ab65bb47fbc5875f2fa87dd78140cfa6eb"}
Oct 30 19:59:09.927996 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:09.926442    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt" event={"ID":"dfc171b1-8278-449b-aea3-e0271b0b525d","Type":"ContainerStarted","Data":"1adecfde6192a44412d4a6c61a955809e943981cc721ed1131690b0179ac3eb9"}
Oct 30 19:59:10.022033 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:10.021998    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:10.041396 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:10.041357    4823 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 30 19:59:10.041680 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:10.041648    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt" podUID="dfc171b1-8278-449b-aea3-e0271b0b525d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 30 19:59:10.087557 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:10.087492    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt" podStartSLOduration=3.087470215 podStartE2EDuration="3.087470215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:09.99913671 +0000 UTC m=+1749.891751070" watchObservedRunningTime="2026-10-30 19:59:10.087470215 +0000 UTC m=+1749.980084540"
Oct 30 19:59:10.895494 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:10.895440    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:59:10.895915 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:10.895876    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc" podUID="44eb96c1-778e-433f-ae45-3a42beed5dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:59:11.315988 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:11.314533    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:11.869885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:11.866365    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:59:11.869885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:11.866437    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc" podUID="44eb96c1-778e-433f-ae45-3a42beed5dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:59:12.012884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.012418    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"]
Oct 30 19:59:12.030365 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.030294    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"
Oct 30 19:59:12.101932 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.101892    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:59:12.103691 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.102895    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:59:12.103691 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.103015    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 19:59:12.103691 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.103115    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:59:12.117185 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.117134    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"]
Oct 30 19:59:12.196703 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.147262    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kz6l\" (UniqueName: \"kubernetes.io/projected/83b10a3f-398f-4228-be67-f92292d43b6f-kube-api-access-9kz6l\") pod \"maas-api-revoke-sub-d2cee2b0-2zfw4\" (UID: \"83b10a3f-398f-4228-be67-f92292d43b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"
Oct 30 19:59:12.196703 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.147331    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b10a3f-398f-4228-be67-f92292d43b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2cee2b0-2zfw4\" (UID: \"83b10a3f-398f-4228-be67-f92292d43b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"
Oct 30 19:59:12.196703 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.147527    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b10a3f-398f-4228-be67-f92292d43b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2cee2b0-2zfw4\" (UID: \"83b10a3f-398f-4228-be67-f92292d43b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"
Oct 30 19:59:12.231847 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.231816    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:12.232981 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.232957    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:12.249706 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.249675    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b10a3f-398f-4228-be67-f92292d43b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2cee2b0-2zfw4\" (UID: \"83b10a3f-398f-4228-be67-f92292d43b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"
Oct 30 19:59:12.250022 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.250003    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kz6l\" (UniqueName: \"kubernetes.io/projected/83b10a3f-398f-4228-be67-f92292d43b6f-kube-api-access-9kz6l\") pod \"maas-api-revoke-sub-d2cee2b0-2zfw4\" (UID: \"83b10a3f-398f-4228-be67-f92292d43b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"
Oct 30 19:59:12.250116 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.250041    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b10a3f-398f-4228-be67-f92292d43b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2cee2b0-2zfw4\" (UID: \"83b10a3f-398f-4228-be67-f92292d43b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"
Oct 30 19:59:12.250385 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.250367    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b10a3f-398f-4228-be67-f92292d43b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2cee2b0-2zfw4\" (UID: \"83b10a3f-398f-4228-be67-f92292d43b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"
Oct 30 19:59:12.324887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.323116    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b10a3f-398f-4228-be67-f92292d43b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2cee2b0-2zfw4\" (UID: \"83b10a3f-398f-4228-be67-f92292d43b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"
Oct 30 19:59:12.349883 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.344973    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kz6l\" (UniqueName: \"kubernetes.io/projected/83b10a3f-398f-4228-be67-f92292d43b6f-kube-api-access-9kz6l\") pod \"maas-api-revoke-sub-d2cee2b0-2zfw4\" (UID: \"83b10a3f-398f-4228-be67-f92292d43b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"
Oct 30 19:59:12.450545 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:12.450127    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"
Oct 30 19:59:13.027101 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:13.015663    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:59:13.027101 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:13.015724    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc" podUID="44eb96c1-778e-433f-ae45-3a42beed5dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:59:13.588935 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:13.585136    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"]
Oct 30 19:59:13.715892 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:13.707267    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4" event={"ID":"83b10a3f-398f-4228-be67-f92292d43b6f","Type":"ContainerStarted","Data":"f3da197a65cd6ddeba5678c1459dffab5e4b3101c965ff54f3f731e183110bd2"}
Oct 30 19:59:13.835500 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:13.835464    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5fa903-ssk7x"]
Oct 30 19:59:13.860082 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:13.859794    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:59:13.860835 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:13.860402    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc" podUID="44eb96c1-778e-433f-ae45-3a42beed5dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:59:13.986624 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:13.986588    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"]
Oct 30 19:59:14.327482 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:14.327442    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b302e4-25dd-4ef6-814b-27075b1c1ca3" path="/var/lib/kubelet/pods/e0b302e4-25dd-4ef6-814b-27075b1c1ca3/volumes"
Oct 30 19:59:14.740725 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:14.740632    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4" event={"ID":"83b10a3f-398f-4228-be67-f92292d43b6f","Type":"ContainerStarted","Data":"1260f8f4a8626e0182c9609d4d70cb0cd0a8c0c90cc889a7603d24e707ce681f"}
Oct 30 19:59:14.816034 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:14.815964    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4" podStartSLOduration=3.8159412919999998 podStartE2EDuration="3.815941292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:14.779392261 +0000 UTC m=+1754.672006590" watchObservedRunningTime="2026-10-30 19:59:14.815941292 +0000 UTC m=+1754.708555615"
Oct 30 19:59:14.921129 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:14.916813    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:14.921129 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:14.917774    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:14.921129 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:14.918683    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 19:59:16.889946 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:16.889011    4823 generic.go:356] "Generic (PLEG): container finished" podID="83b10a3f-398f-4228-be67-f92292d43b6f" containerID="1260f8f4a8626e0182c9609d4d70cb0cd0a8c0c90cc889a7603d24e707ce681f" exitCode=0
Oct 30 19:59:16.890465 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:16.890328    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4" event={"ID":"83b10a3f-398f-4228-be67-f92292d43b6f","Type":"ContainerDied","Data":"1260f8f4a8626e0182c9609d4d70cb0cd0a8c0c90cc889a7603d24e707ce681f"}
Oct 30 19:59:17.122199 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:17.122134    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"]
Oct 30 19:59:17.132341 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:17.132302    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"]
Oct 30 19:59:17.138901 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:17.138853    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8556e06-zpr2k"]
Oct 30 19:59:17.145095 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:17.145001    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d55ccd1-v9gl5"]
Oct 30 19:59:18.125232 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.125197    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"]
Oct 30 19:59:18.137603 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.137571    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70893d69-zhqzc"]
Oct 30 19:59:18.320965 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.320937    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"
Oct 30 19:59:18.323032 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.323002    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e7b0082-74b4-43f3-bf68-d67397801800" path="/var/lib/kubelet/pods/4e7b0082-74b4-43f3-bf68-d67397801800/volumes"
Oct 30 19:59:18.323759 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.323735    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37f565f-b457-42f3-9854-5f94bc8d60f2" path="/var/lib/kubelet/pods/d37f565f-b457-42f3-9854-5f94bc8d60f2/volumes"
Oct 30 19:59:18.324473 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.324450    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e34658db-8f32-48b4-a690-ff7fac03644c" path="/var/lib/kubelet/pods/e34658db-8f32-48b4-a690-ff7fac03644c/volumes"
Oct 30 19:59:18.425954 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.420405    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83b10a3f-398f-4228-be67-f92292d43b6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b10a3f-398f-4228-be67-f92292d43b6f-maas-api-openshift-ca\") pod \"83b10a3f-398f-4228-be67-f92292d43b6f\" (UID: \"83b10a3f-398f-4228-be67-f92292d43b6f\") "
Oct 30 19:59:18.425954 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.420735    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83b10a3f-398f-4228-be67-f92292d43b6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b10a3f-398f-4228-be67-f92292d43b6f-maas-api-xks-ca\") pod \"83b10a3f-398f-4228-be67-f92292d43b6f\" (UID: \"83b10a3f-398f-4228-be67-f92292d43b6f\") "
Oct 30 19:59:18.425954 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.420781    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b10a3f-398f-4228-be67-f92292d43b6f-kube-api-access-9kz6l\" (UniqueName: \"kubernetes.io/projected/83b10a3f-398f-4228-be67-f92292d43b6f-kube-api-access-9kz6l\") pod \"83b10a3f-398f-4228-be67-f92292d43b6f\" (UID: \"83b10a3f-398f-4228-be67-f92292d43b6f\") "
Oct 30 19:59:18.425954 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.425645    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b10a3f-398f-4228-be67-f92292d43b6f-maas-api-openshift-ca" pod "83b10a3f-398f-4228-be67-f92292d43b6f" (UID: "83b10a3f-398f-4228-be67-f92292d43b6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:18.449427 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.449371    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83b10a3f-398f-4228-be67-f92292d43b6f-maas-api-xks-ca" pod "83b10a3f-398f-4228-be67-f92292d43b6f" (UID: "83b10a3f-398f-4228-be67-f92292d43b6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:18.450912 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.450837    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b10a3f-398f-4228-be67-f92292d43b6f-kube-api-access-9kz6l" pod "83b10a3f-398f-4228-be67-f92292d43b6f" (UID: "83b10a3f-398f-4228-be67-f92292d43b6f"). InnerVolumeSpecName "kube-api-access-9kz6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:18.556950 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.538698    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b10a3f-398f-4228-be67-f92292d43b6f-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:18.557226 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.557198    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kz6l\" (UniqueName: \"kubernetes.io/projected/83b10a3f-398f-4228-be67-f92292d43b6f-kube-api-access-9kz6l\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:18.557350 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.557337    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b10a3f-398f-4228-be67-f92292d43b6f-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:18.932517 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.932480    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4" event={"ID":"83b10a3f-398f-4228-be67-f92292d43b6f","Type":"ContainerDied","Data":"f3da197a65cd6ddeba5678c1459dffab5e4b3101c965ff54f3f731e183110bd2"}
Oct 30 19:59:18.932696 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.932539    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3da197a65cd6ddeba5678c1459dffab5e4b3101c965ff54f3f731e183110bd2"
Oct 30 19:59:18.932696 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:18.932590    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"
Oct 30 19:59:21.781364 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:21.780966    4823 patch_prober.go:29] interesting pod/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:59:21.781364 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:21.781038    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x" podUID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:22.920766 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:22.920719    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"]
Oct 30 19:59:22.922101 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:22.922039    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt" podUID="dfc171b1-8278-449b-aea3-e0271b0b525d" containerName="istio-proxy" containerID="cri-o://a49165c07832a8da66130d1d6106d1ab65bb47fbc5875f2fa87dd78140cfa6eb" gracePeriod=30
Oct 30 19:59:23.089139 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:23.089097    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"]
Oct 30 19:59:23.097304 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:23.097260    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d8805a6-vcr9n"]
Oct 30 19:59:24.284361 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:24.284315    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="767ad954-8f7c-481f-9d8a-2f832fe6e07a" path="/var/lib/kubelet/pods/767ad954-8f7c-481f-9d8a-2f832fe6e07a/volumes"
Oct 30 19:59:26.127565 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:26.127526    4823 generic.go:356] "Generic (PLEG): container finished" podID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerID="fc6179186d5c5fe99146518ebf8c18f6151db8b7a762b9337b7b0f4457971e23" exitCode=0
Oct 30 19:59:26.128174 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:26.127585    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x" event={"ID":"d703686a-ccb8-4f4a-8eac-15ce8d774149","Type":"ContainerDied","Data":"fc6179186d5c5fe99146518ebf8c18f6151db8b7a762b9337b7b0f4457971e23"}
Oct 30 19:59:26.128267 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:26.128217    4823 scope.go:122] "RemoveContainer" containerID="fc6179186d5c5fe99146518ebf8c18f6151db8b7a762b9337b7b0f4457971e23"
Oct 30 19:59:26.130746 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:26.130705    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:26.130953 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:26.130802    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:26.130953 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:26.130845    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:27.162814 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:27.151927    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x" event={"ID":"d703686a-ccb8-4f4a-8eac-15ce8d774149","Type":"ContainerStarted","Data":"e8ab576b6cae68f5c196046695047c64ea7ee519485198edb7056dee5e53fc9d"}
Oct 30 19:59:27.162814 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:27.157303    4823 generic.go:356] "Generic (PLEG): container finished" podID="326b0731-4c61-494d-9de4-0ad913784cf1" containerID="66f4a5087cb706b96c414e45c1189fb615fdd35058c63480c8d1b9a669e3350c" exitCode=0
Oct 30 19:59:27.162814 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:27.157599    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj" event={"ID":"326b0731-4c61-494d-9de4-0ad913784cf1","Type":"ContainerDied","Data":"66f4a5087cb706b96c414e45c1189fb615fdd35058c63480c8d1b9a669e3350c"}
Oct 30 19:59:27.162814 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:27.158239    4823 scope.go:122] "RemoveContainer" containerID="66f4a5087cb706b96c414e45c1189fb615fdd35058c63480c8d1b9a669e3350c"
Oct 30 19:59:27.163936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:27.162833    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:27.163936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:27.162957    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:27.163936 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:27.163012    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:27.235590 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:27.235551    4823 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:59:27.235811 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:27.235630    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt" podUID="dfc171b1-8278-449b-aea3-e0271b0b525d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:59:27.858194 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:27.858144    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:59:27.862385 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:27.862344    4823 patch_prober.go:29] interesting pod/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 19:59:27.862674 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:27.862645    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x" podUID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 19:59:28.148549 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.148407    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"]
Oct 30 19:59:28.196332 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.196288    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebad7cfb-whkvk"]
Oct 30 19:59:28.204500 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.204452    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj" event={"ID":"326b0731-4c61-494d-9de4-0ad913784cf1","Type":"ContainerStarted","Data":"f93d2ed16d502d9315fbd5ffde4cfb6804866de5813ba0b705373fed1082c994"}
Oct 30 19:59:28.208462 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.208423    4823 generic.go:356] "Generic (PLEG): container finished" podID="dfc171b1-8278-449b-aea3-e0271b0b525d" containerID="a49165c07832a8da66130d1d6106d1ab65bb47fbc5875f2fa87dd78140cfa6eb" exitCode=0
Oct 30 19:59:28.209643 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.209607    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt" event={"ID":"dfc171b1-8278-449b-aea3-e0271b0b525d","Type":"ContainerDied","Data":"a49165c07832a8da66130d1d6106d1ab65bb47fbc5875f2fa87dd78140cfa6eb"}
Oct 30 19:59:28.473778 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.453349    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:59:28.473778 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.453421    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj" podUID="326b0731-4c61-494d-9de4-0ad913784cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:59:28.528938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.526717    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51289fa4-20a0-4ee8-85ba-f75ff11cb513" path="/var/lib/kubelet/pods/51289fa4-20a0-4ee8-85ba-f75ff11cb513/volumes"
Oct 30 19:59:28.528938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.527515    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:59:28.549380 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:59:28.549337    4823 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc171b1_8278_449b_aea3_e0271b0b525d.slice/crio-conmon-a49165c07832a8da66130d1d6106d1ab65bb47fbc5875f2fa87dd78140cfa6eb.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:59:28.812141 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.812110    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:28.922970 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.908413    4823 patch_prober.go:29] interesting pod/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 19:59:28.922970 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.908503    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x" podUID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 19:59:28.934890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.932985    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"]
Oct 30 19:59:28.939158 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.935109    4823 cpu_manager.go:414] "containerMap: removing container" podUID="83b10a3f-398f-4228-be67-f92292d43b6f" containerName="revoke-keys"
Oct 30 19:59:28.939158 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.935165    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b10a3f-398f-4228-be67-f92292d43b6f" containerName="revoke-keys"
Oct 30 19:59:28.939158 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.935234    4823 cpu_manager.go:414] "containerMap: removing container" podUID="dfc171b1-8278-449b-aea3-e0271b0b525d" containerName="istio-proxy"
Oct 30 19:59:28.939158 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.935242    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc171b1-8278-449b-aea3-e0271b0b525d" containerName="istio-proxy"
Oct 30 19:59:28.939158 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.935604    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b10a3f-398f-4228-be67-f92292d43b6f" containerName="revoke-keys"
Oct 30 19:59:28.939158 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.935639    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc171b1-8278-449b-aea3-e0271b0b525d" containerName="istio-proxy"
Oct 30 19:59:28.941519 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.941492    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:28.956588 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:28.951718    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"]
Oct 30 19:59:29.056939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.051006    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-token\" (UniqueName: \"kubernetes.io/projected/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-token\") pod \"dfc171b1-8278-449b-aea3-e0271b0b525d\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") "
Oct 30 19:59:29.056939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.051134    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-data\") pod \"dfc171b1-8278-449b-aea3-e0271b0b525d\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") "
Oct 30 19:59:29.056939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.051241    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc171b1-8278-449b-aea3-e0271b0b525d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc171b1-8278-449b-aea3-e0271b0b525d-istiod-ca-cert\") pod \"dfc171b1-8278-449b-aea3-e0271b0b525d\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") "
Oct 30 19:59:29.056939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.051357    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-podinfo\") pod \"dfc171b1-8278-449b-aea3-e0271b0b525d\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") "
Oct 30 19:59:29.056939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.051438    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-envoy\") pod \"dfc171b1-8278-449b-aea3-e0271b0b525d\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") "
Oct 30 19:59:29.056939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.051527    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-workload-socket\") pod \"dfc171b1-8278-449b-aea3-e0271b0b525d\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") "
Oct 30 19:59:29.056939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.051694    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-workload-certs\") pod \"dfc171b1-8278-449b-aea3-e0271b0b525d\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") "
Oct 30 19:59:29.056939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.051761    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-credential-socket\") pod \"dfc171b1-8278-449b-aea3-e0271b0b525d\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") "
Oct 30 19:59:29.056939 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.051830    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc171b1-8278-449b-aea3-e0271b0b525d-kube-api-access-mqkrq\" (UniqueName: \"kubernetes.io/projected/dfc171b1-8278-449b-aea3-e0271b0b525d-kube-api-access-mqkrq\") pod \"dfc171b1-8278-449b-aea3-e0271b0b525d\" (UID: \"dfc171b1-8278-449b-aea3-e0271b0b525d\") "
Oct 30 19:59:29.123911 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.078590    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-workload-certs" pod "dfc171b1-8278-449b-aea3-e0271b0b525d" (UID: "dfc171b1-8278-449b-aea3-e0271b0b525d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:29.123911 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.114042    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-data" pod "dfc171b1-8278-449b-aea3-e0271b0b525d" (UID: "dfc171b1-8278-449b-aea3-e0271b0b525d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:29.123911 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.114135    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-workload-socket" pod "dfc171b1-8278-449b-aea3-e0271b0b525d" (UID: "dfc171b1-8278-449b-aea3-e0271b0b525d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:29.123911 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.115099    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc171b1-8278-449b-aea3-e0271b0b525d-istiod-ca-cert" pod "dfc171b1-8278-449b-aea3-e0271b0b525d" (UID: "dfc171b1-8278-449b-aea3-e0271b0b525d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:29.124322 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.124038    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-credential-socket" pod "dfc171b1-8278-449b-aea3-e0271b0b525d" (UID: "dfc171b1-8278-449b-aea3-e0271b0b525d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:29.195902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.167405    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.195902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.167530    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n7jr\" (UniqueName: \"kubernetes.io/projected/be418f6d-e900-479d-bbab-3b94ffba7a7f-kube-api-access-4n7jr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.195902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.167570    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.195902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.167630    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.195902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.167769    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.195902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.167848    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.195902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.167989    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be418f6d-e900-479d-bbab-3b94ffba7a7f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.195902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.168169    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:29.195902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.168188    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:29.195902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.168204    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:29.195902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.168218    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:29.195902 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.168233    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc171b1-8278-449b-aea3-e0271b0b525d-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:29.277264 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.254533    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-token" pod "dfc171b1-8278-449b-aea3-e0271b0b525d" (UID: "dfc171b1-8278-449b-aea3-e0271b0b525d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:29.299047 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.298991    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-envoy" pod "dfc171b1-8278-449b-aea3-e0271b0b525d" (UID: "dfc171b1-8278-449b-aea3-e0271b0b525d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:29.318823 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.311283    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-podinfo" pod "dfc171b1-8278-449b-aea3-e0271b0b525d" (UID: "dfc171b1-8278-449b-aea3-e0271b0b525d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:59:29.318823 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.312708    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.318823 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.312796    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.318823 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.318562    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc171b1-8278-449b-aea3-e0271b0b525d-kube-api-access-mqkrq" pod "dfc171b1-8278-449b-aea3-e0271b0b525d" (UID: "dfc171b1-8278-449b-aea3-e0271b0b525d"). InnerVolumeSpecName "kube-api-access-mqkrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:29.324975 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.323281    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be418f6d-e900-479d-bbab-3b94ffba7a7f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.324975 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.323569    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.324975 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.323679    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n7jr\" (UniqueName: \"kubernetes.io/projected/be418f6d-e900-479d-bbab-3b94ffba7a7f-kube-api-access-4n7jr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.324975 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.323716    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.349887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.346379    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.383885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.356306    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be418f6d-e900-479d-bbab-3b94ffba7a7f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.383885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.356896    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:59:29.383885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.356942    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj" podUID="326b0731-4c61-494d-9de4-0ad913784cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:59:29.383885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.365681    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"]
Oct 30 19:59:29.400236 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.390314    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.403849 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.403807    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"
Oct 30 19:59:29.431934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.423804    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.431934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.423909    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:29.431934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.423931    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:29.431934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.423947    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc171b1-8278-449b-aea3-e0271b0b525d-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:29.431934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.423963    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqkrq\" (UniqueName: \"kubernetes.io/projected/dfc171b1-8278-449b-aea3-e0271b0b525d-kube-api-access-mqkrq\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:29.431934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.424302    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt" event={"ID":"dfc171b1-8278-449b-aea3-e0271b0b525d","Type":"ContainerDied","Data":"1adecfde6192a44412d4a6c61a955809e943981cc721ed1131690b0179ac3eb9"}
Oct 30 19:59:29.431934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.424375    4823 scope.go:122] "RemoveContainer" containerID="a49165c07832a8da66130d1d6106d1ab65bb47fbc5875f2fa87dd78140cfa6eb"
Oct 30 19:59:29.456832 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.456760    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n7jr\" (UniqueName: \"kubernetes.io/projected/be418f6d-e900-479d-bbab-3b94ffba7a7f-kube-api-access-4n7jr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.497031 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.496154    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.500347 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.500264    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.504416 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.502451    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b26sd\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.595942 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.592404    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:29.619672 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.619515    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167a933b-894v5"]
Oct 30 19:59:29.651133 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.651099    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"]
Oct 30 19:59:29.713153 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.713113    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-phxlt"]
Oct 30 19:59:29.929891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.876408    4823 patch_prober.go:29] interesting pod/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 19:59:29.929891 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:29.876485    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x" podUID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 19:59:30.226353 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:30.199224    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"]
Oct 30 19:59:30.245298 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:30.245264    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:30.245477 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:30.245463    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 19:59:30.320415 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:30.320389    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736" path="/var/lib/kubelet/pods/6c9fe8c4-6cfa-4b87-b0f8-a82f0cf4b736/volumes"
Oct 30 19:59:30.321017 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:30.321000    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc171b1-8278-449b-aea3-e0271b0b525d" path="/var/lib/kubelet/pods/dfc171b1-8278-449b-aea3-e0271b0b525d/volumes"
Oct 30 19:59:30.351231 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:30.351200    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:59:30.351414 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:30.351256    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj" podUID="326b0731-4c61-494d-9de4-0ad913784cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:59:30.425073 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:30.423892    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd" event={"ID":"be418f6d-e900-479d-bbab-3b94ffba7a7f","Type":"ContainerStarted","Data":"36833c96bd78fe69de9825b8831c346525dafe2c98e86bce573cb5d3b604061f"}
Oct 30 19:59:30.550764 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:30.549698    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:59:30.872948 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:30.856271    4823 patch_prober.go:29] interesting pod/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 19:59:30.872948 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:30.856329    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x" podUID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 19:59:31.367884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:31.364776    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:59:31.367884 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:31.364920    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj" podUID="326b0731-4c61-494d-9de4-0ad913784cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:59:31.657818 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:31.628572    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd" event={"ID":"be418f6d-e900-479d-bbab-3b94ffba7a7f","Type":"ContainerStarted","Data":"811ab074e2209ec412937a682fb3cef17549565a25e0ed3ba1c6f13a16f5d276"}
Oct 30 19:59:31.657818 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:31.634024    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:31.691952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:31.673296    4823 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-b26sd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:59:31.691952 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:31.673362    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd" podUID="be418f6d-e900-479d-bbab-3b94ffba7a7f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:59:31.750597 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:31.750508    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd" podStartSLOduration=9.750467962 podStartE2EDuration="9.750467962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:31.673157194 +0000 UTC m=+1771.565771519" watchObservedRunningTime="2026-10-30 19:59:31.750467962 +0000 UTC m=+1771.643082287"
Oct 30 19:59:31.859886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:31.858334    4823 patch_prober.go:29] interesting pod/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 19:59:31.859886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:31.858399    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x" podUID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 19:59:32.265978 ip-10-0-138-61 kubenswrapper[4823]: E1030 19:59:32.265336    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 19:59:32.649487 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:32.474985    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:59:32.649487 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:32.475056    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj" podUID="326b0731-4c61-494d-9de4-0ad913784cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:59:32.989076 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:32.988977    4823 patch_prober.go:29] interesting pod/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 19:59:33.011927 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:33.011885    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x" podUID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 19:59:33.238256 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:33.238224    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 19:59:33.447003 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:33.445638    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:59:33.447003 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:33.445700    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj" podUID="326b0731-4c61-494d-9de4-0ad913784cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:59:33.883885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:33.869152    4823 patch_prober.go:29] interesting pod/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 19:59:33.883885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:33.869221    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x" podUID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 19:59:34.034592 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:34.033953    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"]
Oct 30 19:59:34.039602 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:34.039566    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3d2ad17-xdc75"]
Oct 30 19:59:34.301319 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:34.301234    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e0c7d84-3c20-48bb-908c-3624111e0abb" path="/var/lib/kubelet/pods/6e0c7d84-3c20-48bb-908c-3624111e0abb/volumes"
Oct 30 19:59:34.351587 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:34.351543    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:59:34.351775 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:34.351610    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj" podUID="326b0731-4c61-494d-9de4-0ad913784cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:59:35.161607 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:35.161574    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:59:35.164961 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:35.164932    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:59:35.183912 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:35.178217    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 19:59:35.371383 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:35.367890    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:59:35.371383 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:35.367952    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj" podUID="326b0731-4c61-494d-9de4-0ad913784cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:59:36.374904 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:36.364777    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:59:36.475445 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:36.451156    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 19:59:52.053411 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:52.053371    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 19:59:53.336527 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:53.336472    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:53.371379 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:53.371335    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:54.589885 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:54.583513    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:54.741888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:54.741000    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:54.810317 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:54.810269    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:55.047390 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:55.047322    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:59:55.061344 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:55.060368    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:59:55.122099 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:55.122057    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:57.051905 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:57.050016    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:59:57.099888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:57.097074    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:57.124887 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:57.124735    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:57.362948 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:57.353962    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:57.366906 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:57.364703    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:57.409888 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:57.405121    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:57.465890 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:57.464988    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:57.735201 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:57.735100    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:59:57.804201 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:57.804152    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:59:58.052060 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.051951    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:58.053346 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.053317    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:59:58.294886 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.293611    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:58.295199 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.295165    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:59:58.325241 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.295852    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:59:58.347853 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.342559    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:58.401546 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.355990    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:58.474025 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.473962    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:58.480058 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.480006    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:59:58.514402 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.514047    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:59:58.542183 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.542130    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:59:58.544738 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.544708    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:59:58.545069 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.545043    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 19:59:58.545269 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.545255    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:59:58.545602 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.545588    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:59:58.545736 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.545726    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 19:59:58.575346 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.563710    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:59:58.594329 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.594293    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 19:59:58.631934 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.631902    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 19:59:58.637061 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.637028    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 19:59:58.752321 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.747186    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 19:59:58.827157 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.821204    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:58.827157 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.822191    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:59:58.970892 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.968197    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:58.970892 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.970805    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 19:59:58.993139 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:58.971398    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:59.186581 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:59.186525    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:59:59.286910 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:59.202087    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 19:59:59.653881 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:59.653026    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:59:59.825893 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:59.825412    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 19:59:59.828519 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:59.751122    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:00.003938 ip-10-0-138-61 kubenswrapper[4823]: I1030 19:59:59.998842    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:00.141409 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:00.141379    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 20:00:00.202663 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:00.202616    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:00.832450 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:00.832411    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:00:00.833850 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:00.833823    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:00:01.188433 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:01.188400    4823 scope.go:122] "RemoveContainer" containerID="12da6bfeb5e2bfa9145527e1abd38c268ea9f9e99edc41135057ee62a06b3162"
Oct 30 20:00:01.205831 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:01.205769    4823 scope.go:122] "RemoveContainer" containerID="75f5cfb3ed02294fa68421889733393c38cbc7259e2e7499ec83b5757611add1"
Oct 30 20:00:01.218695 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:01.218655    4823 scope.go:122] "RemoveContainer" containerID="228a3b0ec23a290cda6c0e4bc34f7478f9c9ab81b969bad18a68fdb8f4d6250b"
Oct 30 20:00:01.231064 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:01.231041    4823 scope.go:122] "RemoveContainer" containerID="87fcb51cc178d26a0ad86bbb57ce63d2955b652a73fafc1a196cd963b7bc2dab"
Oct 30 20:00:01.243325 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:01.243301    4823 scope.go:122] "RemoveContainer" containerID="ea3f1c47cb45bcb42771bd68e099d6a1da26d862708dafff295d261017b3697a"
Oct 30 20:00:01.280470 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:01.280440    4823 scope.go:122] "RemoveContainer" containerID="564e8a686e8f68ccb1773e823c54b3acd2d58b9959e342fa56eedab3ccf4d1bd"
Oct 30 20:00:01.293217 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:01.293186    4823 scope.go:122] "RemoveContainer" containerID="2699cd3dde34a6e745bad857603f64cc289e3f88c9c01ec4f518af99e514aacd"
Oct 30 20:00:01.305833 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:01.305804    4823 scope.go:122] "RemoveContainer" containerID="0596e9013732de95eaee0ac5011a53be7813f26c950d349cccf81f23ff93d141"
Oct 30 20:00:02.019346 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:00:02.019302    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:00:03.092149 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:03.091111    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"]
Oct 30 20:00:03.117901 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:03.107933    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"]
Oct 30 20:00:03.138552 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:03.138185    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c8d01b-xpctk"]
Oct 30 20:00:03.237947 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:03.237909    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa8a8633-fb9rb"]
Oct 30 20:00:04.284458 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:04.284370    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="714841b4-a1f6-4d03-9d2a-e1ab18481627" path="/var/lib/kubelet/pods/714841b4-a1f6-4d03-9d2a-e1ab18481627/volumes"
Oct 30 20:00:04.285177 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:04.285157    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f20f22cc-d591-4eda-a2ed-5cb640ecd77e" path="/var/lib/kubelet/pods/f20f22cc-d591-4eda-a2ed-5cb640ecd77e/volumes"
Oct 30 20:00:07.863049 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:07.863005    4823 patch_prober.go:29] interesting pod/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:00:07.863573 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:07.863081    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5" podUID="c51ac612-1da2-4152-978a-78726d5ea888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:08.037898 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:08.037819    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"]
Oct 30 20:00:08.043869 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:08.043829    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce5193e-5gmg9"]
Oct 30 20:00:08.282017 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:08.281936    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6efb2f77-0732-43b8-80f4-65cc5ee19ec0" path="/var/lib/kubelet/pods/6efb2f77-0732-43b8-80f4-65cc5ee19ec0/volumes"
Oct 30 20:00:09.546271 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:09.546239    4823 generic.go:356] "Generic (PLEG): container finished" podID="c51ac612-1da2-4152-978a-78726d5ea888" containerID="f08e2a7bdcb16b2064e9884c5ef38f7c4cc455ef3b3e02e890d2675e7664ed84" exitCode=0
Oct 30 20:00:09.546751 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:09.546317    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5" event={"ID":"c51ac612-1da2-4152-978a-78726d5ea888","Type":"ContainerDied","Data":"f08e2a7bdcb16b2064e9884c5ef38f7c4cc455ef3b3e02e890d2675e7664ed84"}
Oct 30 20:00:09.546968 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:09.546948    4823 scope.go:122] "RemoveContainer" containerID="f08e2a7bdcb16b2064e9884c5ef38f7c4cc455ef3b3e02e890d2675e7664ed84"
Oct 30 20:00:09.549567 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:09.549534    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:00:09.549693 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:09.549628    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:00:09.549693 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:09.549682    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:00:10.570984 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:10.562296    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5" event={"ID":"c51ac612-1da2-4152-978a-78726d5ea888","Type":"ContainerStarted","Data":"36eb5db80d327409de404db63914864ff8ac06cffad65c9cc91c31cfdbaeabc0"}
Oct 30 20:00:11.327401 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:11.327370    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 20:00:11.327659 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:11.327633    4823 patch_prober.go:29] interesting pod/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:00:11.327728 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:11.327699    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5" podUID="c51ac612-1da2-4152-978a-78726d5ea888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:00:12.360288 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:12.345012    4823 patch_prober.go:29] interesting pod/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:00:12.360288 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:12.345090    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5" podUID="c51ac612-1da2-4152-978a-78726d5ea888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:00:12.526388 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:12.526351    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"]
Oct 30 20:00:12.526692 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:12.526671    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"]
Oct 30 20:00:12.526923 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:12.526890    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" containerID="cri-o://0ee35e888bf11d1a4662ccac978efd213b6fb3fd02e4f2a27a7b99ce687e8fad" gracePeriod=30
Oct 30 20:00:12.527392 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:12.527366    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m" podUID="5839d58f-f3c6-4adb-a291-9f33b1659f9a" containerName="istio-proxy" containerID="cri-o://ea04bdfed1191bbc27a516123c3045133ae9e461716e5412545bd139f9eddf68" gracePeriod=30
Oct 30 20:00:12.533938 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:12.533899    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"]
Oct 30 20:00:12.534178 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:12.534131    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq" podUID="762ae046-3cc6-4089-891d-5e4fb76ad039" containerName="istio-proxy" containerID="cri-o://b2b1a4f9a5a8e1d82a65e407a495bcd33dd6c600ee75cc5d08fa6439b0a1a889" gracePeriod=30
Oct 30 20:00:13.117792 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:13.117744    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"]
Oct 30 20:00:13.132604 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:13.132565    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34bfc24d-9nxbw"]
Oct 30 20:00:13.362052 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:13.362009    4823 patch_prober.go:29] interesting pod/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:00:13.362747 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:13.362707    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5" podUID="c51ac612-1da2-4152-978a-78726d5ea888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:00:14.145597 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:14.145547    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"]
Oct 30 20:00:14.172238 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:14.164711    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49eeb928-nx7q6"]
Oct 30 20:00:14.287281 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:14.287246    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23661b03-937a-4b72-b668-fb8dc0686ce3" path="/var/lib/kubelet/pods/23661b03-937a-4b72-b668-fb8dc0686ce3/volumes"
Oct 30 20:00:14.287828 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:14.287812    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ed473f-9659-420e-af3d-0f7a6dcc2af6" path="/var/lib/kubelet/pods/76ed473f-9659-420e-af3d-0f7a6dcc2af6/volumes"
Oct 30 20:00:14.328022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:14.327960    4823 patch_prober.go:29] interesting pod/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:00:14.328217 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:14.328026    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5" podUID="c51ac612-1da2-4152-978a-78726d5ea888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:00:15.327611 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:15.327572    4823 patch_prober.go:29] interesting pod/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:00:15.328107 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:15.327634    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5" podUID="c51ac612-1da2-4152-978a-78726d5ea888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:00:16.327020 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:16.326982    4823 patch_prober.go:29] interesting pod/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:00:16.327214 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:16.327057    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5" podUID="c51ac612-1da2-4152-978a-78726d5ea888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:00:16.480301 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:16.480256    4823 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:00:16.480984 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:16.480945    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:00:17.332302 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:17.332269    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 20:00:17.332782 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:17.332767    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 20:00:17.333705 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:17.333681    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 20:00:17.817466 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:17.817294    4823 generic.go:356] "Generic (PLEG): container finished" podID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerID="0ee35e888bf11d1a4662ccac978efd213b6fb3fd02e4f2a27a7b99ce687e8fad" exitCode=0
Oct 30 20:00:17.817466 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:17.817404    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" event={"ID":"330c59cb-17b4-4a1b-8023-c98da7575a34","Type":"ContainerDied","Data":"0ee35e888bf11d1a4662ccac978efd213b6fb3fd02e4f2a27a7b99ce687e8fad"}
Oct 30 20:00:17.820672 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:17.820642    4823 generic.go:356] "Generic (PLEG): container finished" podID="762ae046-3cc6-4089-891d-5e4fb76ad039" containerID="b2b1a4f9a5a8e1d82a65e407a495bcd33dd6c600ee75cc5d08fa6439b0a1a889" exitCode=0
Oct 30 20:00:17.820937 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:17.820727    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq" event={"ID":"762ae046-3cc6-4089-891d-5e4fb76ad039","Type":"ContainerDied","Data":"b2b1a4f9a5a8e1d82a65e407a495bcd33dd6c600ee75cc5d08fa6439b0a1a889"}
Oct 30 20:00:17.825786 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:17.824547    4823 generic.go:356] "Generic (PLEG): container finished" podID="5839d58f-f3c6-4adb-a291-9f33b1659f9a" containerID="ea04bdfed1191bbc27a516123c3045133ae9e461716e5412545bd139f9eddf68" exitCode=0
Oct 30 20:00:17.825786 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:17.825708    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m" event={"ID":"5839d58f-f3c6-4adb-a291-9f33b1659f9a","Type":"ContainerDied","Data":"ea04bdfed1191bbc27a516123c3045133ae9e461716e5412545bd139f9eddf68"}
Oct 30 20:00:17.953019 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:17.952520    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 20:00:17.956970 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:17.956938    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 20:00:18.006970 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.006916    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"]
Oct 30 20:00:18.008345 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.008311    4823 cpu_manager.go:414] "containerMap: removing container" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy"
Oct 30 20:00:18.008345 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.008346    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy"
Oct 30 20:00:18.008538 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.008396    4823 cpu_manager.go:414] "containerMap: removing container" podUID="762ae046-3cc6-4089-891d-5e4fb76ad039" containerName="istio-proxy"
Oct 30 20:00:18.008538 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.008404    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762ae046-3cc6-4089-891d-5e4fb76ad039" containerName="istio-proxy"
Oct 30 20:00:18.008660 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.008643    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" containerName="istio-proxy"
Oct 30 20:00:18.008722 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.008675    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="762ae046-3cc6-4089-891d-5e4fb76ad039" containerName="istio-proxy"
Oct 30 20:00:18.030363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.029602    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-token\" (UniqueName: \"kubernetes.io/projected/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-token\") pod \"762ae046-3cc6-4089-891d-5e4fb76ad039\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") "
Oct 30 20:00:18.030363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.029816    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-envoy\") pod \"330c59cb-17b4-4a1b-8023-c98da7575a34\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") "
Oct 30 20:00:18.030363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.030061    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-workload-certs\") pod \"762ae046-3cc6-4089-891d-5e4fb76ad039\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") "
Oct 30 20:00:18.030363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.030113    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-workload-socket\") pod \"330c59cb-17b4-4a1b-8023-c98da7575a34\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") "
Oct 30 20:00:18.031362 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.031265    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-workload-socket" pod "330c59cb-17b4-4a1b-8023-c98da7575a34" (UID: "330c59cb-17b4-4a1b-8023-c98da7575a34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:18.031362 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.031337    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-workload-certs" pod "762ae046-3cc6-4089-891d-5e4fb76ad039" (UID: "762ae046-3cc6-4089-891d-5e4fb76ad039"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:18.032458 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.032406    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.032458 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.032435    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.033137 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.033110    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-envoy" pod "330c59cb-17b4-4a1b-8023-c98da7575a34" (UID: "330c59cb-17b4-4a1b-8023-c98da7575a34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:18.033475 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.033456    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"]
Oct 30 20:00:18.033563 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.033553    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.035005 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.034975    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-token" pod "762ae046-3cc6-4089-891d-5e4fb76ad039" (UID: "762ae046-3cc6-4089-891d-5e4fb76ad039"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:18.045302 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.045240    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c50e1e-openshift-default-dockercfg-wkgrj\""
Oct 30 20:00:18.059326 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.059298    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 20:00:18.133499 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.133455    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762ae046-3cc6-4089-891d-5e4fb76ad039-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762ae046-3cc6-4089-891d-5e4fb76ad039-istiod-ca-cert\") pod \"762ae046-3cc6-4089-891d-5e4fb76ad039\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") "
Oct 30 20:00:18.133681 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.133551    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-podinfo\") pod \"330c59cb-17b4-4a1b-8023-c98da7575a34\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") "
Oct 30 20:00:18.133681 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.133642    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-data\") pod \"762ae046-3cc6-4089-891d-5e4fb76ad039\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") "
Oct 30 20:00:18.133826 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.133759    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330c59cb-17b4-4a1b-8023-c98da7575a34-kube-api-access-ltdsm\" (UniqueName: \"kubernetes.io/projected/330c59cb-17b4-4a1b-8023-c98da7575a34-kube-api-access-ltdsm\") pod \"330c59cb-17b4-4a1b-8023-c98da7575a34\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") "
Oct 30 20:00:18.133903 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.133824    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-token\" (UniqueName: \"kubernetes.io/projected/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-token\") pod \"330c59cb-17b4-4a1b-8023-c98da7575a34\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") "
Oct 30 20:00:18.133903 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.133893    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-workload-certs\") pod \"330c59cb-17b4-4a1b-8023-c98da7575a34\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") "
Oct 30 20:00:18.134017 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.133965    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762ae046-3cc6-4089-891d-5e4fb76ad039-kube-api-access-qqslp\" (UniqueName: \"kubernetes.io/projected/762ae046-3cc6-4089-891d-5e4fb76ad039-kube-api-access-qqslp\") pod \"762ae046-3cc6-4089-891d-5e4fb76ad039\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") "
Oct 30 20:00:18.134017 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.133973    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762ae046-3cc6-4089-891d-5e4fb76ad039-istiod-ca-cert" pod "762ae046-3cc6-4089-891d-5e4fb76ad039" (UID: "762ae046-3cc6-4089-891d-5e4fb76ad039"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:18.134134 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.134022    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-credential-socket\") pod \"330c59cb-17b4-4a1b-8023-c98da7575a34\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") "
Oct 30 20:00:18.134134 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.134063    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-podinfo\") pod \"762ae046-3cc6-4089-891d-5e4fb76ad039\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") "
Oct 30 20:00:18.134283 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.134260    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-data\") pod \"330c59cb-17b4-4a1b-8023-c98da7575a34\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") "
Oct 30 20:00:18.134495 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.134476    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-credential-socket\") pod \"762ae046-3cc6-4089-891d-5e4fb76ad039\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") "
Oct 30 20:00:18.134682 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.134665    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-envoy\") pod \"762ae046-3cc6-4089-891d-5e4fb76ad039\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") "
Oct 30 20:00:18.134809 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.134790    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-workload-socket\") pod \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") "
Oct 30 20:00:18.134991 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.134972    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-workload-socket\") pod \"762ae046-3cc6-4089-891d-5e4fb76ad039\" (UID: \"762ae046-3cc6-4089-891d-5e4fb76ad039\") "
Oct 30 20:00:18.135133 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.135117    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/330c59cb-17b4-4a1b-8023-c98da7575a34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330c59cb-17b4-4a1b-8023-c98da7575a34-istiod-ca-cert\") pod \"330c59cb-17b4-4a1b-8023-c98da7575a34\" (UID: \"330c59cb-17b4-4a1b-8023-c98da7575a34\") "
Oct 30 20:00:18.135313 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.135295    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-data\") pod \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") "
Oct 30 20:00:18.135941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.135912    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-envoy\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.136367 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.136308    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fsdn\" (UniqueName: \"kubernetes.io/projected/7b162ce0-cc0c-4f76-802d-9175c518f6df-kube-api-access-8fsdn\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.136554 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.136535    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-token\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.136728 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.134338    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-data" pod "762ae046-3cc6-4089-891d-5e4fb76ad039" (UID: "762ae046-3cc6-4089-891d-5e4fb76ad039"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:18.136804 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.134533    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-credential-socket" pod "330c59cb-17b4-4a1b-8023-c98da7575a34" (UID: "330c59cb-17b4-4a1b-8023-c98da7575a34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:18.136804 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.134732    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-workload-certs" pod "330c59cb-17b4-4a1b-8023-c98da7575a34" (UID: "330c59cb-17b4-4a1b-8023-c98da7575a34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:18.136804 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.135055    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-data" pod "330c59cb-17b4-4a1b-8023-c98da7575a34" (UID: "330c59cb-17b4-4a1b-8023-c98da7575a34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:18.136804 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.135332    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-credential-socket" pod "762ae046-3cc6-4089-891d-5e4fb76ad039" (UID: "762ae046-3cc6-4089-891d-5e4fb76ad039"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:18.136804 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.135700    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-workload-socket" pod "762ae046-3cc6-4089-891d-5e4fb76ad039" (UID: "762ae046-3cc6-4089-891d-5e4fb76ad039"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:18.136804 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.135971    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-workload-socket" pod "5839d58f-f3c6-4adb-a291-9f33b1659f9a" (UID: "5839d58f-f3c6-4adb-a291-9f33b1659f9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:18.136804 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.136198    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-data" pod "5839d58f-f3c6-4adb-a291-9f33b1659f9a" (UID: "5839d58f-f3c6-4adb-a291-9f33b1659f9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:18.137197 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.136512    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330c59cb-17b4-4a1b-8023-c98da7575a34-istiod-ca-cert" pod "330c59cb-17b4-4a1b-8023-c98da7575a34" (UID: "330c59cb-17b4-4a1b-8023-c98da7575a34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:18.137197 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.136603    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-podinfo" pod "330c59cb-17b4-4a1b-8023-c98da7575a34" (UID: "330c59cb-17b4-4a1b-8023-c98da7575a34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:00:18.137301 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.137262    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-podinfo" pod "762ae046-3cc6-4089-891d-5e4fb76ad039" (UID: "762ae046-3cc6-4089-891d-5e4fb76ad039"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:00:18.137301 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.137286    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330c59cb-17b4-4a1b-8023-c98da7575a34-kube-api-access-ltdsm" pod "330c59cb-17b4-4a1b-8023-c98da7575a34" (UID: "330c59cb-17b4-4a1b-8023-c98da7575a34"). InnerVolumeSpecName "kube-api-access-ltdsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:18.137492 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.137469    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-workload-certs\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.137661 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.137643    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b162ce0-cc0c-4f76-802d-9175c518f6df-istiod-ca-cert\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.137750 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.137732    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-credential-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.137936 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.137847    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-podinfo\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.138082 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138065    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-workload-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.138258 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138240    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-data\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.138366 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138327    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762ae046-3cc6-4089-891d-5e4fb76ad039-kube-api-access-qqslp" pod "762ae046-3cc6-4089-891d-5e4fb76ad039" (UID: "762ae046-3cc6-4089-891d-5e4fb76ad039"). InnerVolumeSpecName "kube-api-access-qqslp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:18.138815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138585    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762ae046-3cc6-4089-891d-5e4fb76ad039-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.138815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138612    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.138815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138626    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.138815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138641    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.138815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138654    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltdsm\" (UniqueName: \"kubernetes.io/projected/330c59cb-17b4-4a1b-8023-c98da7575a34-kube-api-access-ltdsm\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.138815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138669    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.138815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138683    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.138815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138697    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqslp\" (UniqueName: \"kubernetes.io/projected/762ae046-3cc6-4089-891d-5e4fb76ad039-kube-api-access-qqslp\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.138815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138711    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.138815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138725    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.138815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138738    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.138815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138751    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.138815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138764    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.138815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138777    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.138815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138787    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330c59cb-17b4-4a1b-8023-c98da7575a34-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.138815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138796    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.139494 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.138839    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-envoy" pod "762ae046-3cc6-4089-891d-5e4fb76ad039" (UID: "762ae046-3cc6-4089-891d-5e4fb76ad039"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:18.139494 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.139385    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-token" pod "330c59cb-17b4-4a1b-8023-c98da7575a34" (UID: "330c59cb-17b4-4a1b-8023-c98da7575a34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:18.239630 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.239595    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-credential-socket\") pod \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") "
Oct 30 20:00:18.239843 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.239680    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-token\" (UniqueName: \"kubernetes.io/projected/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-token\") pod \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") "
Oct 30 20:00:18.239951 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.239924    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-credential-socket" pod "5839d58f-f3c6-4adb-a291-9f33b1659f9a" (UID: "5839d58f-f3c6-4adb-a291-9f33b1659f9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:18.240025 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.239934    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5839d58f-f3c6-4adb-a291-9f33b1659f9a-kube-api-access-6hq6c\" (UniqueName: \"kubernetes.io/projected/5839d58f-f3c6-4adb-a291-9f33b1659f9a-kube-api-access-6hq6c\") pod \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") "
Oct 30 20:00:18.240092 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.240048    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-envoy\") pod \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") "
Oct 30 20:00:18.240152 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.240105    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istiod-ca-cert\") pod \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") "
Oct 30 20:00:18.240152 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.240141    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-podinfo\") pod \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") "
Oct 30 20:00:18.240243 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.240171    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-workload-certs\") pod \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\" (UID: \"5839d58f-f3c6-4adb-a291-9f33b1659f9a\") "
Oct 30 20:00:18.240526 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.240497    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-workload-certs" pod "5839d58f-f3c6-4adb-a291-9f33b1659f9a" (UID: "5839d58f-f3c6-4adb-a291-9f33b1659f9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:18.240646 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.240549    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istiod-ca-cert" pod "5839d58f-f3c6-4adb-a291-9f33b1659f9a" (UID: "5839d58f-f3c6-4adb-a291-9f33b1659f9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:18.240646 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.240567    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-credential-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.240646 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.240604    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-podinfo\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.240646 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.240632    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-workload-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.240783 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.240677    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-data\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.240970 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.240944    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-envoy\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.241109 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.241081    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-workload-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.241166 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.241106    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-credential-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.241222 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.241181    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fsdn\" (UniqueName: \"kubernetes.io/projected/7b162ce0-cc0c-4f76-802d-9175c518f6df-kube-api-access-8fsdn\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.241285 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.241244    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-token\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.241379 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.241361    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-workload-certs\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.241437 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.241374    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-data\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.241437 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.241425    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b162ce0-cc0c-4f76-802d-9175c518f6df-istiod-ca-cert\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.241587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.241566    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762ae046-3cc6-4089-891d-5e4fb76ad039-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.241661 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.241595    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.241661 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.241612    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.241661 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.241627    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330c59cb-17b4-4a1b-8023-c98da7575a34-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.241661 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.241636    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-workload-certs\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.241890 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.241642    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.242449 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.242427    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b162ce0-cc0c-4f76-802d-9175c518f6df-istiod-ca-cert\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.242717 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.242695    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-envoy" pod "5839d58f-f3c6-4adb-a291-9f33b1659f9a" (UID: "5839d58f-f3c6-4adb-a291-9f33b1659f9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:18.242810 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.242733    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-token" pod "5839d58f-f3c6-4adb-a291-9f33b1659f9a" (UID: "5839d58f-f3c6-4adb-a291-9f33b1659f9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:18.243225 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.243202    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-envoy\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.243339 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.243289    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5839d58f-f3c6-4adb-a291-9f33b1659f9a-kube-api-access-6hq6c" pod "5839d58f-f3c6-4adb-a291-9f33b1659f9a" (UID: "5839d58f-f3c6-4adb-a291-9f33b1659f9a"). InnerVolumeSpecName "kube-api-access-6hq6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:18.243933 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.243916    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-podinfo\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.244118 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.244094    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-podinfo" pod "5839d58f-f3c6-4adb-a291-9f33b1659f9a" (UID: "5839d58f-f3c6-4adb-a291-9f33b1659f9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:00:18.260967 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.260935    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-token\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.261703 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.261678    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fsdn\" (UniqueName: \"kubernetes.io/projected/7b162ce0-cc0c-4f76-802d-9175c518f6df-kube-api-access-8fsdn\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.344261 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.344221    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hq6c\" (UniqueName: \"kubernetes.io/projected/5839d58f-f3c6-4adb-a291-9f33b1659f9a-kube-api-access-6hq6c\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.344261 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.344259    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.344469 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.344277    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.344469 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.344293    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5839d58f-f3c6-4adb-a291-9f33b1659f9a-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.356230 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.356197    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:18.662356 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:00:18.662325    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b162ce0_cc0c_4f76_802d_9175c518f6df.slice/crio-f4da11d6b23f11b9455710125768a993a1e53c9f7cc55b83dd1e970b8d4e811f WatchSource:0}: Error finding container f4da11d6b23f11b9455710125768a993a1e53c9f7cc55b83dd1e970b8d4e811f: Status 404 returned error can't find the container with id f4da11d6b23f11b9455710125768a993a1e53c9f7cc55b83dd1e970b8d4e811f
Oct 30 20:00:18.664261 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.664227    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"]
Oct 30 20:00:18.665827 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.665788    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:00:18.666053 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.665889    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:00:18.666053 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.665935    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:00:18.839899 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.839836    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2" event={"ID":"7b162ce0-cc0c-4f76-802d-9175c518f6df","Type":"ContainerStarted","Data":"b8ed26a374750bd61410b7fcf446c3189fe44053e8a8cc56e1a1ea9565a260b0"}
Oct 30 20:00:18.840376 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.839961    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2" event={"ID":"7b162ce0-cc0c-4f76-802d-9175c518f6df","Type":"ContainerStarted","Data":"f4da11d6b23f11b9455710125768a993a1e53c9f7cc55b83dd1e970b8d4e811f"}
Oct 30 20:00:18.842582 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.842024    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"
Oct 30 20:00:18.842582 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.842031    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw" event={"ID":"330c59cb-17b4-4a1b-8023-c98da7575a34","Type":"ContainerDied","Data":"54bc2afc4187114a017af79bbb3d90cddfb2ea3931b2116f67ad1747f8b423e0"}
Oct 30 20:00:18.842582 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.842108    4823 scope.go:122] "RemoveContainer" containerID="0ee35e888bf11d1a4662ccac978efd213b6fb3fd02e4f2a27a7b99ce687e8fad"
Oct 30 20:00:18.844317 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.844285    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq" event={"ID":"762ae046-3cc6-4089-891d-5e4fb76ad039","Type":"ContainerDied","Data":"b8f8ee27da5998c88642288dd80972a9a78f5e3542ddd4c22eb58c8eb21052cf"}
Oct 30 20:00:18.844317 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.844309    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"
Oct 30 20:00:18.846622 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.846467    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m" event={"ID":"5839d58f-f3c6-4adb-a291-9f33b1659f9a","Type":"ContainerDied","Data":"35897b5987093446425af04b859fb064127cc1a972f0ad96fd5caab1f58a2067"}
Oct 30 20:00:18.846622 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.846487    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"
Oct 30 20:00:18.858660 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.858637    4823 scope.go:122] "RemoveContainer" containerID="b2b1a4f9a5a8e1d82a65e407a495bcd33dd6c600ee75cc5d08fa6439b0a1a889"
Oct 30 20:00:18.866730 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.866665    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2" podStartSLOduration=26.866645047 podStartE2EDuration="26.866645047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:18.86280746 +0000 UTC m=+1818.755421785" watchObservedRunningTime="2026-10-30 20:00:18.866645047 +0000 UTC m=+1818.759259373"
Oct 30 20:00:18.875020 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.874905    4823 scope.go:122] "RemoveContainer" containerID="ea04bdfed1191bbc27a516123c3045133ae9e461716e5412545bd139f9eddf68"
Oct 30 20:00:18.904884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.887000    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"]
Oct 30 20:00:18.904884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.896343    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-zk82m"]
Oct 30 20:00:18.932005 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.931942    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"]
Oct 30 20:00:18.938941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.938909    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-rbvrw"]
Oct 30 20:00:18.945363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.945329    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"]
Oct 30 20:00:18.950440 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:18.950413    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-j5mrq"]
Oct 30 20:00:19.357491 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:19.357448    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:19.364207 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:19.364176    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:19.860141 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:19.860111    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:19.861345 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:19.861324    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:00:20.282344 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:20.282251    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330c59cb-17b4-4a1b-8023-c98da7575a34" path="/var/lib/kubelet/pods/330c59cb-17b4-4a1b-8023-c98da7575a34/volumes"
Oct 30 20:00:20.283051 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:20.283032    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5839d58f-f3c6-4adb-a291-9f33b1659f9a" path="/var/lib/kubelet/pods/5839d58f-f3c6-4adb-a291-9f33b1659f9a/volumes"
Oct 30 20:00:20.283653 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:20.283639    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762ae046-3cc6-4089-891d-5e4fb76ad039" path="/var/lib/kubelet/pods/762ae046-3cc6-4089-891d-5e4fb76ad039/volumes"
Oct 30 20:00:29.029735 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.029685    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"]
Oct 30 20:00:29.035330 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.035294    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f70995d-2p867"]
Oct 30 20:00:29.472852 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.472813    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"]
Oct 30 20:00:29.473778 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.473757    4823 cpu_manager.go:414] "containerMap: removing container" podUID="5839d58f-f3c6-4adb-a291-9f33b1659f9a" containerName="istio-proxy"
Oct 30 20:00:29.473778 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.473778    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5839d58f-f3c6-4adb-a291-9f33b1659f9a" containerName="istio-proxy"
Oct 30 20:00:29.474007 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.473991    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="5839d58f-f3c6-4adb-a291-9f33b1659f9a" containerName="istio-proxy"
Oct 30 20:00:29.478379 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.478350    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:00:29.487791 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.487747    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"]
Oct 30 20:00:29.488477 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.488122    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47lpr\""
Oct 30 20:00:29.496109 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.493129    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-35deec-serving-cert\""
Oct 30 20:00:29.496109 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.493678    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:00:29.496109 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.493769    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:00:29.523259 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.523194    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 20:00:29.534901 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.534842    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2911d92-5384-4bbf-b875-c846a7df7459-metrics-certs\") pod \"maas-api-e2e-api-a-35deec-7d56cdf4-g26hj\" (UID: \"c2911d92-5384-4bbf-b875-c846a7df7459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:00:29.534901 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.534903    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g27r\" (UniqueName: \"kubernetes.io/projected/c2911d92-5384-4bbf-b875-c846a7df7459-kube-api-access-9g27r\") pod \"maas-api-e2e-api-a-35deec-7d56cdf4-g26hj\" (UID: \"c2911d92-5384-4bbf-b875-c846a7df7459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:00:29.535709 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.535681    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2911d92-5384-4bbf-b875-c846a7df7459-maas-api-tls\") pod \"maas-api-e2e-api-a-35deec-7d56cdf4-g26hj\" (UID: \"c2911d92-5384-4bbf-b875-c846a7df7459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:00:29.535847 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.535818    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2911d92-5384-4bbf-b875-c846a7df7459-openshift-service-ca\") pod \"maas-api-e2e-api-a-35deec-7d56cdf4-g26hj\" (UID: \"c2911d92-5384-4bbf-b875-c846a7df7459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:00:29.648888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.648309    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2911d92-5384-4bbf-b875-c846a7df7459-maas-api-tls\") pod \"maas-api-e2e-api-a-35deec-7d56cdf4-g26hj\" (UID: \"c2911d92-5384-4bbf-b875-c846a7df7459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:00:29.648888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.648385    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2911d92-5384-4bbf-b875-c846a7df7459-openshift-service-ca\") pod \"maas-api-e2e-api-a-35deec-7d56cdf4-g26hj\" (UID: \"c2911d92-5384-4bbf-b875-c846a7df7459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:00:29.648888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.648548    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2911d92-5384-4bbf-b875-c846a7df7459-metrics-certs\") pod \"maas-api-e2e-api-a-35deec-7d56cdf4-g26hj\" (UID: \"c2911d92-5384-4bbf-b875-c846a7df7459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:00:29.648888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.648576    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g27r\" (UniqueName: \"kubernetes.io/projected/c2911d92-5384-4bbf-b875-c846a7df7459-kube-api-access-9g27r\") pod \"maas-api-e2e-api-a-35deec-7d56cdf4-g26hj\" (UID: \"c2911d92-5384-4bbf-b875-c846a7df7459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:00:29.654756 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.654717    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2911d92-5384-4bbf-b875-c846a7df7459-openshift-service-ca\") pod \"maas-api-e2e-api-a-35deec-7d56cdf4-g26hj\" (UID: \"c2911d92-5384-4bbf-b875-c846a7df7459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:00:29.678155 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.676494    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2911d92-5384-4bbf-b875-c846a7df7459-maas-api-tls\") pod \"maas-api-e2e-api-a-35deec-7d56cdf4-g26hj\" (UID: \"c2911d92-5384-4bbf-b875-c846a7df7459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:00:29.724434 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.719380    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2911d92-5384-4bbf-b875-c846a7df7459-metrics-certs\") pod \"maas-api-e2e-api-a-35deec-7d56cdf4-g26hj\" (UID: \"c2911d92-5384-4bbf-b875-c846a7df7459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:00:29.733194 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.724720    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g27r\" (UniqueName: \"kubernetes.io/projected/c2911d92-5384-4bbf-b875-c846a7df7459-kube-api-access-9g27r\") pod \"maas-api-e2e-api-a-35deec-7d56cdf4-g26hj\" (UID: \"c2911d92-5384-4bbf-b875-c846a7df7459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:00:29.813188 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:29.813150    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:00:30.122659 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:00:30.122623    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2911d92_5384_4bbf_b875_c846a7df7459.slice/crio-d153770f1c9e98bcd08be6904357cef15c6f18966f111f5cd099bf20c34a51d6 WatchSource:0}: Error finding container d153770f1c9e98bcd08be6904357cef15c6f18966f111f5cd099bf20c34a51d6: Status 404 returned error can't find the container with id d153770f1c9e98bcd08be6904357cef15c6f18966f111f5cd099bf20c34a51d6
Oct 30 20:00:30.123034 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:30.122796    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"]
Oct 30 20:00:30.124324 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:30.124302    4823 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:00:30.238542 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:30.238500    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj" event={"ID":"c2911d92-5384-4bbf-b875-c846a7df7459","Type":"ContainerStarted","Data":"d153770f1c9e98bcd08be6904357cef15c6f18966f111f5cd099bf20c34a51d6"}
Oct 30 20:00:30.289667 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:30.289631    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f10e1fd-8a47-4d0a-8eb0-85d6763c7515" path="/var/lib/kubelet/pods/2f10e1fd-8a47-4d0a-8eb0-85d6763c7515/volumes"
Oct 30 20:00:32.060358 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:00:32.060225    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:00:33.030324 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:33.030273    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"]
Oct 30 20:00:33.036378 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:33.036344    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f40b9632-9nhn9"]
Oct 30 20:00:33.291695 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:33.291602    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj" event={"ID":"c2911d92-5384-4bbf-b875-c846a7df7459","Type":"ContainerStarted","Data":"872ff9a2d3a8a20bf0ac497fdd92484a6f3f07940f34d614a5247068992318c7"}
Oct 30 20:00:33.291695 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:33.291686    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:00:33.313518 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:33.313460    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj" podStartSLOduration=1.829651597 podStartE2EDuration="4.31344529s" totalImagesPullingTime="2.483793693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:33.310700808 +0000 UTC m=+1833.203315133" watchObservedRunningTime="2026-10-30 20:00:33.31344529 +0000 UTC m=+1833.206059613"
Oct 30 20:00:34.038213 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:34.038171    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"]
Oct 30 20:00:34.043368 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:34.043339    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb581435-f2skc"]
Oct 30 20:00:34.281935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:34.281900    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d56c80-79bf-4e01-968f-3beab184830f" path="/var/lib/kubelet/pods/09d56c80-79bf-4e01-968f-3beab184830f/volumes"
Oct 30 20:00:34.282534 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:34.282516    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ba9b3b-5381-47f8-bf01-b4f9fa104713" path="/var/lib/kubelet/pods/a2ba9b3b-5381-47f8-bf01-b4f9fa104713/volumes"
Oct 30 20:00:39.038453 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:39.038405    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"]
Oct 30 20:00:39.043614 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:39.043579    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805a1f72-lmfnd"]
Oct 30 20:00:39.307893 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:39.307790    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:00:40.050176 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:40.050129    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:00:40.374226 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:40.374181    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5aabcbc-69dc-4da8-8589-5305dd2f77d0" path="/var/lib/kubelet/pods/c5aabcbc-69dc-4da8-8589-5305dd2f77d0/volumes"
Oct 30 20:00:41.698560 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:41.698511    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"]
Oct 30 20:00:41.699392 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:41.699343    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz" podUID="0ad80bfe-3826-4dd0-bb18-c7710c865123" containerName="istio-proxy" containerID="cri-o://dc8b7d1f9784a99c668afa8f1a2a864de50fe773d927728b4b826bfcb3e4a86d" gracePeriod=30
Oct 30 20:00:47.441042 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.441011    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 20:00:47.536781 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.536683    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-data\") pod \"0ad80bfe-3826-4dd0-bb18-c7710c865123\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") "
Oct 30 20:00:47.536781 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.536750    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-credential-socket\") pod \"0ad80bfe-3826-4dd0-bb18-c7710c865123\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") "
Oct 30 20:00:47.536781 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.536780    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-token\" (UniqueName: \"kubernetes.io/projected/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-token\") pod \"0ad80bfe-3826-4dd0-bb18-c7710c865123\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") "
Oct 30 20:00:47.537126 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.536908    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-envoy\") pod \"0ad80bfe-3826-4dd0-bb18-c7710c865123\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") "
Oct 30 20:00:47.537126 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.537088    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-data" pod "0ad80bfe-3826-4dd0-bb18-c7710c865123" (UID: "0ad80bfe-3826-4dd0-bb18-c7710c865123"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:47.537230 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.537131    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-credential-socket" pod "0ad80bfe-3826-4dd0-bb18-c7710c865123" (UID: "0ad80bfe-3826-4dd0-bb18-c7710c865123"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:47.537280 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.537229    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad80bfe-3826-4dd0-bb18-c7710c865123-kube-api-access-h2pbw\" (UniqueName: \"kubernetes.io/projected/0ad80bfe-3826-4dd0-bb18-c7710c865123-kube-api-access-h2pbw\") pod \"0ad80bfe-3826-4dd0-bb18-c7710c865123\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") "
Oct 30 20:00:47.537280 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.537256    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-workload-certs\") pod \"0ad80bfe-3826-4dd0-bb18-c7710c865123\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") "
Oct 30 20:00:47.537387 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.537310    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-podinfo\") pod \"0ad80bfe-3826-4dd0-bb18-c7710c865123\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") "
Oct 30 20:00:47.537387 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.537343    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-workload-socket\") pod \"0ad80bfe-3826-4dd0-bb18-c7710c865123\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") "
Oct 30 20:00:47.537485 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.537409    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad80bfe-3826-4dd0-bb18-c7710c865123-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ad80bfe-3826-4dd0-bb18-c7710c865123-istiod-ca-cert\") pod \"0ad80bfe-3826-4dd0-bb18-c7710c865123\" (UID: \"0ad80bfe-3826-4dd0-bb18-c7710c865123\") "
Oct 30 20:00:47.537543 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.537518    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-workload-certs" pod "0ad80bfe-3826-4dd0-bb18-c7710c865123" (UID: "0ad80bfe-3826-4dd0-bb18-c7710c865123"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:47.537898 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.537854    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-workload-socket" pod "0ad80bfe-3826-4dd0-bb18-c7710c865123" (UID: "0ad80bfe-3826-4dd0-bb18-c7710c865123"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:47.538536 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.538151    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad80bfe-3826-4dd0-bb18-c7710c865123-istiod-ca-cert" pod "0ad80bfe-3826-4dd0-bb18-c7710c865123" (UID: "0ad80bfe-3826-4dd0-bb18-c7710c865123"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:47.538536 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.538399    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.538536 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.538424    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.538536 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.538439    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.538536 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.538455    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.538536 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.538469    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ad80bfe-3826-4dd0-bb18-c7710c865123-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.539694 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.539664    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad80bfe-3826-4dd0-bb18-c7710c865123-kube-api-access-h2pbw" pod "0ad80bfe-3826-4dd0-bb18-c7710c865123" (UID: "0ad80bfe-3826-4dd0-bb18-c7710c865123"). InnerVolumeSpecName "kube-api-access-h2pbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:47.539788 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.539696    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-podinfo" pod "0ad80bfe-3826-4dd0-bb18-c7710c865123" (UID: "0ad80bfe-3826-4dd0-bb18-c7710c865123"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:00:47.540257 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.540232    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-token" pod "0ad80bfe-3826-4dd0-bb18-c7710c865123" (UID: "0ad80bfe-3826-4dd0-bb18-c7710c865123"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:47.541526 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.541500    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-envoy" pod "0ad80bfe-3826-4dd0-bb18-c7710c865123" (UID: "0ad80bfe-3826-4dd0-bb18-c7710c865123"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:47.641588 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.641071    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.641588 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.641131    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.641588 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.641150    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2pbw\" (UniqueName: \"kubernetes.io/projected/0ad80bfe-3826-4dd0-bb18-c7710c865123-kube-api-access-h2pbw\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.641588 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.641176    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ad80bfe-3826-4dd0-bb18-c7710c865123-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.650620 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.650585    4823 generic.go:356] "Generic (PLEG): container finished" podID="0ad80bfe-3826-4dd0-bb18-c7710c865123" containerID="dc8b7d1f9784a99c668afa8f1a2a864de50fe773d927728b4b826bfcb3e4a86d" exitCode=0
Oct 30 20:00:47.650779 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.650633    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"
Oct 30 20:00:47.650779 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.650675    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz" event={"ID":"0ad80bfe-3826-4dd0-bb18-c7710c865123","Type":"ContainerDied","Data":"dc8b7d1f9784a99c668afa8f1a2a864de50fe773d927728b4b826bfcb3e4a86d"}
Oct 30 20:00:47.650779 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.650734    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz" event={"ID":"0ad80bfe-3826-4dd0-bb18-c7710c865123","Type":"ContainerDied","Data":"0e6e670f3855266428efe1743f213f16b1bfcc502a38717a405cd6541e94c479"}
Oct 30 20:00:47.650779 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.650764    4823 scope.go:122] "RemoveContainer" containerID="dc8b7d1f9784a99c668afa8f1a2a864de50fe773d927728b4b826bfcb3e4a86d"
Oct 30 20:00:47.665141 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.665111    4823 scope.go:122] "RemoveContainer" containerID="dc8b7d1f9784a99c668afa8f1a2a864de50fe773d927728b4b826bfcb3e4a86d"
Oct 30 20:00:47.665532 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:00:47.665509    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc8b7d1f9784a99c668afa8f1a2a864de50fe773d927728b4b826bfcb3e4a86d\": container with ID starting with dc8b7d1f9784a99c668afa8f1a2a864de50fe773d927728b4b826bfcb3e4a86d not found: ID does not exist" containerID="dc8b7d1f9784a99c668afa8f1a2a864de50fe773d927728b4b826bfcb3e4a86d"
Oct 30 20:00:47.665614 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.665547    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc8b7d1f9784a99c668afa8f1a2a864de50fe773d927728b4b826bfcb3e4a86d"} err="failed to get container status \"dc8b7d1f9784a99c668afa8f1a2a864de50fe773d927728b4b826bfcb3e4a86d\": rpc error: code = NotFound desc = could not find container \"dc8b7d1f9784a99c668afa8f1a2a864de50fe773d927728b4b826bfcb3e4a86d\": container with ID starting with dc8b7d1f9784a99c668afa8f1a2a864de50fe773d927728b4b826bfcb3e4a86d not found: ID does not exist"
Oct 30 20:00:47.682413 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.682378    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"]
Oct 30 20:00:47.691510 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:47.691476    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-d2thz"]
Oct 30 20:00:48.282710 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:48.282671    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad80bfe-3826-4dd0-bb18-c7710c865123" path="/var/lib/kubelet/pods/0ad80bfe-3826-4dd0-bb18-c7710c865123/volumes"
Oct 30 20:00:52.796259 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:52.796217    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"]
Oct 30 20:00:52.797316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:52.797290    4823 cpu_manager.go:414] "containerMap: removing container" podUID="0ad80bfe-3826-4dd0-bb18-c7710c865123" containerName="istio-proxy"
Oct 30 20:00:52.797316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:52.797315    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad80bfe-3826-4dd0-bb18-c7710c865123" containerName="istio-proxy"
Oct 30 20:00:52.797565 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:52.797549    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad80bfe-3826-4dd0-bb18-c7710c865123" containerName="istio-proxy"
Oct 30 20:00:52.801383 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:52.801360    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"
Oct 30 20:00:52.808116 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:52.808080    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:00:52.808269 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:52.808195    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:00:52.813584 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:52.813550    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"]
Oct 30 20:00:52.920161 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:52.920114    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8ffe44-35d9-40c2-b767-b5610445757b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-060bf0e1-ghq2f\" (UID: \"fd8ffe44-35d9-40c2-b767-b5610445757b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"
Oct 30 20:00:52.920393 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:52.920307    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdsr4\" (UniqueName: \"kubernetes.io/projected/fd8ffe44-35d9-40c2-b767-b5610445757b-kube-api-access-vdsr4\") pod \"maas-api-revoke-sub-060bf0e1-ghq2f\" (UID: \"fd8ffe44-35d9-40c2-b767-b5610445757b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"
Oct 30 20:00:52.920676 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:52.920655    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8ffe44-35d9-40c2-b767-b5610445757b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-060bf0e1-ghq2f\" (UID: \"fd8ffe44-35d9-40c2-b767-b5610445757b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"
Oct 30 20:00:53.039989 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.039510    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8ffe44-35d9-40c2-b767-b5610445757b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-060bf0e1-ghq2f\" (UID: \"fd8ffe44-35d9-40c2-b767-b5610445757b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"
Oct 30 20:00:53.039989 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.039742    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8ffe44-35d9-40c2-b767-b5610445757b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-060bf0e1-ghq2f\" (UID: \"fd8ffe44-35d9-40c2-b767-b5610445757b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"
Oct 30 20:00:53.039989 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.039837    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdsr4\" (UniqueName: \"kubernetes.io/projected/fd8ffe44-35d9-40c2-b767-b5610445757b-kube-api-access-vdsr4\") pod \"maas-api-revoke-sub-060bf0e1-ghq2f\" (UID: \"fd8ffe44-35d9-40c2-b767-b5610445757b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"
Oct 30 20:00:53.067949 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.065957    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8ffe44-35d9-40c2-b767-b5610445757b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-060bf0e1-ghq2f\" (UID: \"fd8ffe44-35d9-40c2-b767-b5610445757b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"
Oct 30 20:00:53.087285 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.085696    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8ffe44-35d9-40c2-b767-b5610445757b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-060bf0e1-ghq2f\" (UID: \"fd8ffe44-35d9-40c2-b767-b5610445757b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"
Oct 30 20:00:53.120885 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.110428    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdsr4\" (UniqueName: \"kubernetes.io/projected/fd8ffe44-35d9-40c2-b767-b5610445757b-kube-api-access-vdsr4\") pod \"maas-api-revoke-sub-060bf0e1-ghq2f\" (UID: \"fd8ffe44-35d9-40c2-b767-b5610445757b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"
Oct 30 20:00:53.130524 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.123754    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"
Oct 30 20:00:53.334943 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.334154    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"]
Oct 30 20:00:53.342883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.342698    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.367889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.353448    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80073467-436a-4974-b535-170e24af7092-istiod-ca-cert\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.367889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.353568    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-istio-envoy\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.367889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.353642    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80073467-436a-4974-b535-170e24af7092-istio-token\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.367889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.353740    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-workload-certs\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.367889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.353806    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrkj5\" (UniqueName: \"kubernetes.io/projected/80073467-436a-4974-b535-170e24af7092-kube-api-access-wrkj5\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.367889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.354568    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-credential-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.367889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.354693    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-workload-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.367889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.354741    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80073467-436a-4974-b535-170e24af7092-istio-podinfo\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.367889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.354768    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-istio-data\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.380548 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.380511    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dd1b04-openshift-default-dockercfg-dhzsm\""
Oct 30 20:00:53.479888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.459835    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80073467-436a-4974-b535-170e24af7092-istiod-ca-cert\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.479888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.469025    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-istio-envoy\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.479888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.469155    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80073467-436a-4974-b535-170e24af7092-istio-token\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.479888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.469309    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-workload-certs\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.479888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.469376    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrkj5\" (UniqueName: \"kubernetes.io/projected/80073467-436a-4974-b535-170e24af7092-kube-api-access-wrkj5\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.479888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.469474    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-credential-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.479888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.469596    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-workload-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.479888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.469643    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80073467-436a-4974-b535-170e24af7092-istio-podinfo\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.479888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.469674    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-istio-data\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.479888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.470277    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-workload-certs\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.479888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.470618    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-workload-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.479888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.471174    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-credential-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.479888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.471829    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-istio-data\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.479888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.473124    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"]
Oct 30 20:00:53.489809 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.489112    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80073467-436a-4974-b535-170e24af7092-istio-podinfo\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.510660 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.510561    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrkj5\" (UniqueName: \"kubernetes.io/projected/80073467-436a-4974-b535-170e24af7092-kube-api-access-wrkj5\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.581660 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.581600    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80073467-436a-4974-b535-170e24af7092-istiod-ca-cert\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.659706 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.642322    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80073467-436a-4974-b535-170e24af7092-istio-token\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.659706 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.642594    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-istio-envoy\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp\" (UID: \"80073467-436a-4974-b535-170e24af7092\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.768851 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.768803    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:53.928517 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:53.928473    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"]
Oct 30 20:00:53.928987 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:00:53.928477    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8ffe44_35d9_40c2_b767_b5610445757b.slice/crio-b1ea6e6a79aaa1d2de7fd0a36a0c7126d46fb697c01283e5b9457978e431b24c WatchSource:0}: Error finding container b1ea6e6a79aaa1d2de7fd0a36a0c7126d46fb697c01283e5b9457978e431b24c: Status 404 returned error can't find the container with id b1ea6e6a79aaa1d2de7fd0a36a0c7126d46fb697c01283e5b9457978e431b24c
Oct 30 20:00:54.076639 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:54.076607    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"]
Oct 30 20:00:54.155567 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:00:54.155523    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80073467_436a_4974_b535_170e24af7092.slice/crio-8be0bdf468adad2c210276f17ac28d3aa1f927279802a3d32b4a0f6315ae112b WatchSource:0}: Error finding container 8be0bdf468adad2c210276f17ac28d3aa1f927279802a3d32b4a0f6315ae112b: Status 404 returned error can't find the container with id 8be0bdf468adad2c210276f17ac28d3aa1f927279802a3d32b4a0f6315ae112b
Oct 30 20:00:54.159060 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:54.159023    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:00:54.159151 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:54.159099    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:00:54.159151 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:54.159137    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:00:54.852441 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:54.852403    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp" event={"ID":"80073467-436a-4974-b535-170e24af7092","Type":"ContainerStarted","Data":"a9d5f284d96f4002c4a72df39d17c2be2af5a9a1350b2be90103d079afbfb3bd"}
Oct 30 20:00:54.852654 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:54.852464    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp" event={"ID":"80073467-436a-4974-b535-170e24af7092","Type":"ContainerStarted","Data":"8be0bdf468adad2c210276f17ac28d3aa1f927279802a3d32b4a0f6315ae112b"}
Oct 30 20:00:54.854578 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:54.854548    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f" event={"ID":"fd8ffe44-35d9-40c2-b767-b5610445757b","Type":"ContainerStarted","Data":"46b16e1d73100bc60333796285b985f790a0af1c3c2e07849d86f4989b797ddf"}
Oct 30 20:00:54.854712 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:54.854604    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f" event={"ID":"fd8ffe44-35d9-40c2-b767-b5610445757b","Type":"ContainerStarted","Data":"b1ea6e6a79aaa1d2de7fd0a36a0c7126d46fb697c01283e5b9457978e431b24c"}
Oct 30 20:00:54.885570 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:54.885504    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp" podStartSLOduration=1.8854836719999999 podStartE2EDuration="1.885483672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:54.876469 +0000 UTC m=+1854.769083322" watchObservedRunningTime="2026-10-30 20:00:54.885483672 +0000 UTC m=+1854.778097997"
Oct 30 20:00:54.917218 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:54.912752    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f" podStartSLOduration=2.912726456 podStartE2EDuration="2.912726456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:54.905159779 +0000 UTC m=+1854.797774103" watchObservedRunningTime="2026-10-30 20:00:54.912726456 +0000 UTC m=+1854.805340782"
Oct 30 20:00:55.772237 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:55.772204    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:00:55.773108 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:55.773077    4823 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 20:00:55.774057 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:55.774024    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp" podUID="80073467-436a-4974-b535-170e24af7092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 20:00:55.934883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:55.932945    4823 generic.go:356] "Generic (PLEG): container finished" podID="fd8ffe44-35d9-40c2-b767-b5610445757b" containerID="46b16e1d73100bc60333796285b985f790a0af1c3c2e07849d86f4989b797ddf" exitCode=0
Oct 30 20:00:55.934883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:55.933620    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f" event={"ID":"fd8ffe44-35d9-40c2-b767-b5610445757b","Type":"ContainerDied","Data":"46b16e1d73100bc60333796285b985f790a0af1c3c2e07849d86f4989b797ddf"}
Oct 30 20:00:56.779300 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:56.779265    4823 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 20:00:56.779894 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:56.779837    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp" podUID="80073467-436a-4974-b535-170e24af7092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 20:00:57.789284 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:57.789241    4823 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 20:00:57.789284 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:57.789291    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp" podUID="80073467-436a-4974-b535-170e24af7092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 20:00:57.896882 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:57.890295    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"
Oct 30 20:00:57.993897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:57.977656    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd8ffe44-35d9-40c2-b767-b5610445757b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8ffe44-35d9-40c2-b767-b5610445757b-maas-api-xks-ca\") pod \"fd8ffe44-35d9-40c2-b767-b5610445757b\" (UID: \"fd8ffe44-35d9-40c2-b767-b5610445757b\") "
Oct 30 20:00:57.993897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:57.977803    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8ffe44-35d9-40c2-b767-b5610445757b-kube-api-access-vdsr4\" (UniqueName: \"kubernetes.io/projected/fd8ffe44-35d9-40c2-b767-b5610445757b-kube-api-access-vdsr4\") pod \"fd8ffe44-35d9-40c2-b767-b5610445757b\" (UID: \"fd8ffe44-35d9-40c2-b767-b5610445757b\") "
Oct 30 20:00:57.993897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:57.977874    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8ffe44-35d9-40c2-b767-b5610445757b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8ffe44-35d9-40c2-b767-b5610445757b-maas-api-openshift-ca\") pod \"fd8ffe44-35d9-40c2-b767-b5610445757b\" (UID: \"fd8ffe44-35d9-40c2-b767-b5610445757b\") "
Oct 30 20:00:57.993897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:57.988124    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8ffe44-35d9-40c2-b767-b5610445757b-maas-api-openshift-ca" pod "fd8ffe44-35d9-40c2-b767-b5610445757b" (UID: "fd8ffe44-35d9-40c2-b767-b5610445757b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:58.005934 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:58.004817    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8ffe44-35d9-40c2-b767-b5610445757b-kube-api-access-vdsr4" pod "fd8ffe44-35d9-40c2-b767-b5610445757b" (UID: "fd8ffe44-35d9-40c2-b767-b5610445757b"). InnerVolumeSpecName "kube-api-access-vdsr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:58.007608 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:58.007535    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd8ffe44-35d9-40c2-b767-b5610445757b-maas-api-xks-ca" pod "fd8ffe44-35d9-40c2-b767-b5610445757b" (UID: "fd8ffe44-35d9-40c2-b767-b5610445757b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:58.089977 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:58.085683    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8ffe44-35d9-40c2-b767-b5610445757b-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:58.089977 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:58.085752    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdsr4\" (UniqueName: \"kubernetes.io/projected/fd8ffe44-35d9-40c2-b767-b5610445757b-kube-api-access-vdsr4\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:58.089977 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:58.085769    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8ffe44-35d9-40c2-b767-b5610445757b-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:58.335803 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:58.335776    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"
Oct 30 20:00:58.335803 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:58.335788    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f" event={"ID":"fd8ffe44-35d9-40c2-b767-b5610445757b","Type":"ContainerDied","Data":"b1ea6e6a79aaa1d2de7fd0a36a0c7126d46fb697c01283e5b9457978e431b24c"}
Oct 30 20:00:58.336055 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:58.335845    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1ea6e6a79aaa1d2de7fd0a36a0c7126d46fb697c01283e5b9457978e431b24c"
Oct 30 20:00:58.776893 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:58.775365    4823 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 20:00:58.776893 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:58.775431    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp" podUID="80073467-436a-4974-b535-170e24af7092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 20:00:59.775739 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:59.775697    4823 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 20:00:59.776560 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:00:59.776478    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp" podUID="80073467-436a-4974-b535-170e24af7092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 20:01:00.775302 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:00.775261    4823 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 20:01:00.776212 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:00.776173    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp" podUID="80073467-436a-4974-b535-170e24af7092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 20:01:01.442980 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:01.442942    4823 scope.go:122] "RemoveContainer" containerID="c309d0360d3813eef7253df26c65b6ef3f2347a5140a6efd92b210eda6871fc9"
Oct 30 20:01:01.481847 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:01.481820    4823 scope.go:122] "RemoveContainer" containerID="95201873bc7906360856f1555267fa164c89e195a5abd2044036bada28bdc1ab"
Oct 30 20:01:01.494032 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:01.494003    4823 scope.go:122] "RemoveContainer" containerID="694a9d22e525edb29b6a79cca847343673d4e16e7660bdb48d3e67356f6c1697"
Oct 30 20:01:01.506118 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:01.506094    4823 scope.go:122] "RemoveContainer" containerID="d63b7fddd7578d23f6fafbd211d8d6e56b4fb0460dd1504ba0440ac237a9d21f"
Oct 30 20:01:01.518486 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:01.518460    4823 scope.go:122] "RemoveContainer" containerID="cd3873a36219c74a78b62a5e2fba580ea853d764848b410c259ad35d27451823"
Oct 30 20:01:01.535968 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:01.535945    4823 scope.go:122] "RemoveContainer" containerID="5283506b7e33e28a93eab2bb1d377bac34df5c486da5b428e1317d6fe3de4ac4"
Oct 30 20:01:01.548081 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:01.548055    4823 scope.go:122] "RemoveContainer" containerID="bb815c521bf1aea059dc6cc4dbb99aeac9ab3000b12861f5b14035818e082aec"
Oct 30 20:01:01.568494 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:01.568455    4823 scope.go:122] "RemoveContainer" containerID="138ac8357d065bbe6a1eb05df0c58b9906ec00a7547352d2f6ceac27c02090e9"
Oct 30 20:01:01.580584 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:01.580547    4823 scope.go:122] "RemoveContainer" containerID="ebf55bb429f9afadf14fa34934e19c1ab994e3f7439c7b3263ecfd93b2cedf2f"
Oct 30 20:01:01.605522 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:01.605500    4823 scope.go:122] "RemoveContainer" containerID="0075beafc1fb515ecd335c7a58bf2d51b0d4660bf5b59f0e16028c18e19369cf"
Oct 30 20:01:01.854297 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:01.854261    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:01:01.854804 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:01.854791    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:01:01.855797 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:01.855775    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:01:02.045881 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:01:02.045758    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:01:02.290530 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.290498    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"]
Oct 30 20:01:02.291413 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.291391    4823 cpu_manager.go:414] "containerMap: removing container" podUID="fd8ffe44-35d9-40c2-b767-b5610445757b" containerName="revoke-keys"
Oct 30 20:01:02.291473 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.291413    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8ffe44-35d9-40c2-b767-b5610445757b" containerName="revoke-keys"
Oct 30 20:01:02.291609 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.291598    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8ffe44-35d9-40c2-b767-b5610445757b" containerName="revoke-keys"
Oct 30 20:01:02.297565 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.297493    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"
Oct 30 20:01:02.301325 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.301298    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:02.301582 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.301540    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:01:02.307205 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.307043    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"]
Oct 30 20:01:02.423472 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.423436    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs8xg\" (UniqueName: \"kubernetes.io/projected/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-kube-api-access-zs8xg\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j\" (UID: \"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"
Oct 30 20:01:02.423642 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.423522    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j\" (UID: \"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"
Oct 30 20:01:02.423765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.423749    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j\" (UID: \"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"
Oct 30 20:01:02.525509 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.525466    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs8xg\" (UniqueName: \"kubernetes.io/projected/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-kube-api-access-zs8xg\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j\" (UID: \"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"
Oct 30 20:01:02.525726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.525537    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j\" (UID: \"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"
Oct 30 20:01:02.525805 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.525783    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j\" (UID: \"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"
Oct 30 20:01:02.526424 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.526400    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j\" (UID: \"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"
Oct 30 20:01:02.528136 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.528111    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j\" (UID: \"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"
Oct 30 20:01:02.568195 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.561243    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs8xg\" (UniqueName: \"kubernetes.io/projected/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-kube-api-access-zs8xg\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j\" (UID: \"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"
Oct 30 20:01:02.629280 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.629237    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"
Oct 30 20:01:02.952123 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:01:02.952081    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d1a2c2_5642_4e1d_9de9_edc32176a1f8.slice/crio-a45a5879406cb550153d7643d67f3eab5651dc89f7d19e263b346209505ee3d6 WatchSource:0}: Error finding container a45a5879406cb550153d7643d67f3eab5651dc89f7d19e263b346209505ee3d6: Status 404 returned error can't find the container with id a45a5879406cb550153d7643d67f3eab5651dc89f7d19e263b346209505ee3d6
Oct 30 20:01:02.956929 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:02.956895    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"]
Oct 30 20:01:03.606352 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:03.598045    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j" event={"ID":"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8","Type":"ContainerStarted","Data":"a45a5879406cb550153d7643d67f3eab5651dc89f7d19e263b346209505ee3d6"}
Oct 30 20:01:04.608811 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:04.608777    4823 generic.go:356] "Generic (PLEG): container finished" podID="a1d1a2c2-5642-4e1d-9de9-edc32176a1f8" containerID="ebd738347a61c2eade6e612b012d8de11364f4b82baace8757a4fdf0471a2943" exitCode=0
Oct 30 20:01:04.609256 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:04.608876    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j" event={"ID":"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8","Type":"ContainerDied","Data":"ebd738347a61c2eade6e612b012d8de11364f4b82baace8757a4fdf0471a2943"}
Oct 30 20:01:05.480838 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:05.480781    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"]
Oct 30 20:01:05.485679 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:05.485640    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x" podUID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerName="istio-proxy" containerID="cri-o://e8ab576b6cae68f5c196046695047c64ea7ee519485198edb7056dee5e53fc9d" gracePeriod=30
Oct 30 20:01:06.013718 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:05.998921    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"]
Oct 30 20:01:06.047887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.031234    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"
Oct 30 20:01:06.090887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.086950    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"]
Oct 30 20:01:06.185892 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.174818    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx8k2\" (UniqueName: \"kubernetes.io/projected/c5f003b6-482e-4099-94af-1853828bdd2f-kube-api-access-bx8k2\") pod \"maas-api-revoke-keys-e2e-api-b-c4c072-knps9\" (UID: \"c5f003b6-482e-4099-94af-1853828bdd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"
Oct 30 20:01:06.185892 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.174937    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f003b6-482e-4099-94af-1853828bdd2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4c072-knps9\" (UID: \"c5f003b6-482e-4099-94af-1853828bdd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"
Oct 30 20:01:06.185892 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.175013    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f003b6-482e-4099-94af-1853828bdd2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4c072-knps9\" (UID: \"c5f003b6-482e-4099-94af-1853828bdd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"
Oct 30 20:01:06.277825 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.277619    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f003b6-482e-4099-94af-1853828bdd2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4c072-knps9\" (UID: \"c5f003b6-482e-4099-94af-1853828bdd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"
Oct 30 20:01:06.287894 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.286980    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f003b6-482e-4099-94af-1853828bdd2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4c072-knps9\" (UID: \"c5f003b6-482e-4099-94af-1853828bdd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"
Oct 30 20:01:06.287894 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.287564    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8k2\" (UniqueName: \"kubernetes.io/projected/c5f003b6-482e-4099-94af-1853828bdd2f-kube-api-access-bx8k2\") pod \"maas-api-revoke-keys-e2e-api-b-c4c072-knps9\" (UID: \"c5f003b6-482e-4099-94af-1853828bdd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"
Oct 30 20:01:06.288198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.288165    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f003b6-482e-4099-94af-1853828bdd2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4c072-knps9\" (UID: \"c5f003b6-482e-4099-94af-1853828bdd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"
Oct 30 20:01:06.305481 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.305444    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f003b6-482e-4099-94af-1853828bdd2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c4c072-knps9\" (UID: \"c5f003b6-482e-4099-94af-1853828bdd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"
Oct 30 20:01:06.329640 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.329602    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8k2\" (UniqueName: \"kubernetes.io/projected/c5f003b6-482e-4099-94af-1853828bdd2f-kube-api-access-bx8k2\") pod \"maas-api-revoke-keys-e2e-api-b-c4c072-knps9\" (UID: \"c5f003b6-482e-4099-94af-1853828bdd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"
Oct 30 20:01:06.416197 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.416163    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"
Oct 30 20:01:06.419196 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.419174    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"
Oct 30 20:01:06.491760 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.491725    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-kube-api-access-zs8xg\" (UniqueName: \"kubernetes.io/projected/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-kube-api-access-zs8xg\") pod \"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8\" (UID: \"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8\") "
Oct 30 20:01:06.491951 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.491896    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-maas-api-openshift-ca\") pod \"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8\" (UID: \"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8\") "
Oct 30 20:01:06.492040 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.492023    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-maas-api-xks-ca\") pod \"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8\" (UID: \"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8\") "
Oct 30 20:01:06.494815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.494783    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-maas-api-openshift-ca" pod "a1d1a2c2-5642-4e1d-9de9-edc32176a1f8" (UID: "a1d1a2c2-5642-4e1d-9de9-edc32176a1f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:06.495158 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.495054    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-kube-api-access-zs8xg" pod "a1d1a2c2-5642-4e1d-9de9-edc32176a1f8" (UID: "a1d1a2c2-5642-4e1d-9de9-edc32176a1f8"). InnerVolumeSpecName "kube-api-access-zs8xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:06.503920 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.503629    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-maas-api-xks-ca" pod "a1d1a2c2-5642-4e1d-9de9-edc32176a1f8" (UID: "a1d1a2c2-5642-4e1d-9de9-edc32176a1f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:06.596224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.596132    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:06.596224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.596173    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs8xg\" (UniqueName: \"kubernetes.io/projected/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-kube-api-access-zs8xg\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:06.596224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.596183    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:06.823829 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:01:06.823785    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f003b6_482e_4099_94af_1853828bdd2f.slice/crio-97e4e92309c928809c267d53a2a771bb54efcd752f5aa7702b937c40c778832e WatchSource:0}: Error finding container 97e4e92309c928809c267d53a2a771bb54efcd752f5aa7702b937c40c778832e: Status 404 returned error can't find the container with id 97e4e92309c928809c267d53a2a771bb54efcd752f5aa7702b937c40c778832e
Oct 30 20:01:06.825639 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.825606    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j" event={"ID":"a1d1a2c2-5642-4e1d-9de9-edc32176a1f8","Type":"ContainerDied","Data":"a45a5879406cb550153d7643d67f3eab5651dc89f7d19e263b346209505ee3d6"}
Oct 30 20:01:06.825731 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.825673    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a45a5879406cb550153d7643d67f3eab5651dc89f7d19e263b346209505ee3d6"
Oct 30 20:01:06.825773 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.825746    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"
Oct 30 20:01:06.831325 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:06.831297    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"]
Oct 30 20:01:07.960255 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:07.960103    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9" event={"ID":"c5f003b6-482e-4099-94af-1853828bdd2f","Type":"ContainerStarted","Data":"3e5cc4a0732bbaccf43aea27ebfe935878593c2eb7205f1b3c0f306d6d14b5e6"}
Oct 30 20:01:07.960255 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:07.960171    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9" event={"ID":"c5f003b6-482e-4099-94af-1853828bdd2f","Type":"ContainerStarted","Data":"97e4e92309c928809c267d53a2a771bb54efcd752f5aa7702b937c40c778832e"}
Oct 30 20:01:08.007888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:08.001820    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9" podStartSLOduration=3.001798242 podStartE2EDuration="3.001798242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:08.001629023 +0000 UTC m=+1867.894243349" watchObservedRunningTime="2026-10-30 20:01:08.001798242 +0000 UTC m=+1867.894412567"
Oct 30 20:01:08.971600 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:08.971558    4823 generic.go:356] "Generic (PLEG): container finished" podID="c5f003b6-482e-4099-94af-1853828bdd2f" containerID="3e5cc4a0732bbaccf43aea27ebfe935878593c2eb7205f1b3c0f306d6d14b5e6" exitCode=0
Oct 30 20:01:08.972088 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:08.971639    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9" event={"ID":"c5f003b6-482e-4099-94af-1853828bdd2f","Type":"ContainerDied","Data":"3e5cc4a0732bbaccf43aea27ebfe935878593c2eb7205f1b3c0f306d6d14b5e6"}
Oct 30 20:01:10.185481 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.185453    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"
Oct 30 20:01:10.308274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.308179    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5f003b6-482e-4099-94af-1853828bdd2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f003b6-482e-4099-94af-1853828bdd2f-maas-api-openshift-ca\") pod \"c5f003b6-482e-4099-94af-1853828bdd2f\" (UID: \"c5f003b6-482e-4099-94af-1853828bdd2f\") "
Oct 30 20:01:10.308274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.308250    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5f003b6-482e-4099-94af-1853828bdd2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f003b6-482e-4099-94af-1853828bdd2f-maas-api-xks-ca\") pod \"c5f003b6-482e-4099-94af-1853828bdd2f\" (UID: \"c5f003b6-482e-4099-94af-1853828bdd2f\") "
Oct 30 20:01:10.308501 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.308287    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f003b6-482e-4099-94af-1853828bdd2f-kube-api-access-bx8k2\" (UniqueName: \"kubernetes.io/projected/c5f003b6-482e-4099-94af-1853828bdd2f-kube-api-access-bx8k2\") pod \"c5f003b6-482e-4099-94af-1853828bdd2f\" (UID: \"c5f003b6-482e-4099-94af-1853828bdd2f\") "
Oct 30 20:01:10.308660 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.308624    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f003b6-482e-4099-94af-1853828bdd2f-maas-api-openshift-ca" pod "c5f003b6-482e-4099-94af-1853828bdd2f" (UID: "c5f003b6-482e-4099-94af-1853828bdd2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:10.309457 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.309434    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f003b6-482e-4099-94af-1853828bdd2f-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:10.310616 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.310595    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5f003b6-482e-4099-94af-1853828bdd2f-maas-api-xks-ca" pod "c5f003b6-482e-4099-94af-1853828bdd2f" (UID: "c5f003b6-482e-4099-94af-1853828bdd2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:10.310708 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.310673    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f003b6-482e-4099-94af-1853828bdd2f-kube-api-access-bx8k2" pod "c5f003b6-482e-4099-94af-1853828bdd2f" (UID: "c5f003b6-482e-4099-94af-1853828bdd2f"). InnerVolumeSpecName "kube-api-access-bx8k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:10.411463 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.411432    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f003b6-482e-4099-94af-1853828bdd2f-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:10.411463 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.411460    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx8k2\" (UniqueName: \"kubernetes.io/projected/c5f003b6-482e-4099-94af-1853828bdd2f-kube-api-access-bx8k2\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:10.709434 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.709403    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 20:01:10.818508 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.818465    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d703686a-ccb8-4f4a-8eac-15ce8d774149-kube-api-access-s5p6h\" (UniqueName: \"kubernetes.io/projected/d703686a-ccb8-4f4a-8eac-15ce8d774149-kube-api-access-s5p6h\") pod \"d703686a-ccb8-4f4a-8eac-15ce8d774149\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") "
Oct 30 20:01:10.818738 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.818658    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-podinfo\") pod \"d703686a-ccb8-4f4a-8eac-15ce8d774149\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") "
Oct 30 20:01:10.818814 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.818751    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-credential-socket\") pod \"d703686a-ccb8-4f4a-8eac-15ce8d774149\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") "
Oct 30 20:01:10.818814 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.818796    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-workload-socket\") pod \"d703686a-ccb8-4f4a-8eac-15ce8d774149\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") "
Oct 30 20:01:10.818987 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.818962    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-workload-certs\") pod \"d703686a-ccb8-4f4a-8eac-15ce8d774149\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") "
Oct 30 20:01:10.819123 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.819105    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-data\") pod \"d703686a-ccb8-4f4a-8eac-15ce8d774149\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") "
Oct 30 20:01:10.819266 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.819233    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-credential-socket" pod "d703686a-ccb8-4f4a-8eac-15ce8d774149" (UID: "d703686a-ccb8-4f4a-8eac-15ce8d774149"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:10.819357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.819332    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-token\" (UniqueName: \"kubernetes.io/projected/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-token\") pod \"d703686a-ccb8-4f4a-8eac-15ce8d774149\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") "
Oct 30 20:01:10.819419 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.819382    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d703686a-ccb8-4f4a-8eac-15ce8d774149-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d703686a-ccb8-4f4a-8eac-15ce8d774149-istiod-ca-cert\") pod \"d703686a-ccb8-4f4a-8eac-15ce8d774149\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") "
Oct 30 20:01:10.819478 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.819416    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-envoy\") pod \"d703686a-ccb8-4f4a-8eac-15ce8d774149\" (UID: \"d703686a-ccb8-4f4a-8eac-15ce8d774149\") "
Oct 30 20:01:10.820173 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.820149    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-data" pod "d703686a-ccb8-4f4a-8eac-15ce8d774149" (UID: "d703686a-ccb8-4f4a-8eac-15ce8d774149"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:10.820387 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.820362    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:10.820387 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.820386    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:10.820612 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.820594    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-workload-socket" pod "d703686a-ccb8-4f4a-8eac-15ce8d774149" (UID: "d703686a-ccb8-4f4a-8eac-15ce8d774149"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:10.821184 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.821143    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d703686a-ccb8-4f4a-8eac-15ce8d774149-istiod-ca-cert" pod "d703686a-ccb8-4f4a-8eac-15ce8d774149" (UID: "d703686a-ccb8-4f4a-8eac-15ce8d774149"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:10.821535 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.821498    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-workload-certs" pod "d703686a-ccb8-4f4a-8eac-15ce8d774149" (UID: "d703686a-ccb8-4f4a-8eac-15ce8d774149"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:10.822280 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.822260    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-podinfo" pod "d703686a-ccb8-4f4a-8eac-15ce8d774149" (UID: "d703686a-ccb8-4f4a-8eac-15ce8d774149"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:10.822551 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.822530    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d703686a-ccb8-4f4a-8eac-15ce8d774149-kube-api-access-s5p6h" pod "d703686a-ccb8-4f4a-8eac-15ce8d774149" (UID: "d703686a-ccb8-4f4a-8eac-15ce8d774149"). InnerVolumeSpecName "kube-api-access-s5p6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:10.824164 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.824135    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-envoy" pod "d703686a-ccb8-4f4a-8eac-15ce8d774149" (UID: "d703686a-ccb8-4f4a-8eac-15ce8d774149"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:10.824548 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.824521    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-token" pod "d703686a-ccb8-4f4a-8eac-15ce8d774149" (UID: "d703686a-ccb8-4f4a-8eac-15ce8d774149"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:10.952651 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.952062    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:10.952651 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.952102    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:10.952651 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.952117    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d703686a-ccb8-4f4a-8eac-15ce8d774149-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:10.952651 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.952129    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:10.952651 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.952142    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5p6h\" (UniqueName: \"kubernetes.io/projected/d703686a-ccb8-4f4a-8eac-15ce8d774149-kube-api-access-s5p6h\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:10.952651 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.952155    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d703686a-ccb8-4f4a-8eac-15ce8d774149-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:10.952651 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.952169    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d703686a-ccb8-4f4a-8eac-15ce8d774149-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:10.992268 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.992229    4823 generic.go:356] "Generic (PLEG): container finished" podID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerID="e8ab576b6cae68f5c196046695047c64ea7ee519485198edb7056dee5e53fc9d" exitCode=0
Oct 30 20:01:10.992425 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.992288    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"
Oct 30 20:01:10.992425 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.992306    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x" event={"ID":"d703686a-ccb8-4f4a-8eac-15ce8d774149","Type":"ContainerDied","Data":"e8ab576b6cae68f5c196046695047c64ea7ee519485198edb7056dee5e53fc9d"}
Oct 30 20:01:10.992425 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.992371    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x" event={"ID":"d703686a-ccb8-4f4a-8eac-15ce8d774149","Type":"ContainerDied","Data":"4642f056183949317e3dfeab028c0f7f91ef6c2c79273c5e2b2ca1958d01de3a"}
Oct 30 20:01:10.992425 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.992397    4823 scope.go:122] "RemoveContainer" containerID="e8ab576b6cae68f5c196046695047c64ea7ee519485198edb7056dee5e53fc9d"
Oct 30 20:01:10.994431 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.994412    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"
Oct 30 20:01:10.994542 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.994422    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9" event={"ID":"c5f003b6-482e-4099-94af-1853828bdd2f","Type":"ContainerDied","Data":"97e4e92309c928809c267d53a2a771bb54efcd752f5aa7702b937c40c778832e"}
Oct 30 20:01:10.994542 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:10.994472    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97e4e92309c928809c267d53a2a771bb54efcd752f5aa7702b937c40c778832e"
Oct 30 20:01:11.006018 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:11.005994    4823 scope.go:122] "RemoveContainer" containerID="fc6179186d5c5fe99146518ebf8c18f6151db8b7a762b9337b7b0f4457971e23"
Oct 30 20:01:11.017900 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:11.017879    4823 scope.go:122] "RemoveContainer" containerID="e8ab576b6cae68f5c196046695047c64ea7ee519485198edb7056dee5e53fc9d"
Oct 30 20:01:11.018244 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:01:11.018224    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ab576b6cae68f5c196046695047c64ea7ee519485198edb7056dee5e53fc9d\": container with ID starting with e8ab576b6cae68f5c196046695047c64ea7ee519485198edb7056dee5e53fc9d not found: ID does not exist" containerID="e8ab576b6cae68f5c196046695047c64ea7ee519485198edb7056dee5e53fc9d"
Oct 30 20:01:11.018297 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:11.018255    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ab576b6cae68f5c196046695047c64ea7ee519485198edb7056dee5e53fc9d"} err="failed to get container status \"e8ab576b6cae68f5c196046695047c64ea7ee519485198edb7056dee5e53fc9d\": rpc error: code = NotFound desc = could not find container \"e8ab576b6cae68f5c196046695047c64ea7ee519485198edb7056dee5e53fc9d\": container with ID starting with e8ab576b6cae68f5c196046695047c64ea7ee519485198edb7056dee5e53fc9d not found: ID does not exist"
Oct 30 20:01:11.018297 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:11.018275    4823 scope.go:122] "RemoveContainer" containerID="fc6179186d5c5fe99146518ebf8c18f6151db8b7a762b9337b7b0f4457971e23"
Oct 30 20:01:11.018582 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:01:11.018561    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc6179186d5c5fe99146518ebf8c18f6151db8b7a762b9337b7b0f4457971e23\": container with ID starting with fc6179186d5c5fe99146518ebf8c18f6151db8b7a762b9337b7b0f4457971e23 not found: ID does not exist" containerID="fc6179186d5c5fe99146518ebf8c18f6151db8b7a762b9337b7b0f4457971e23"
Oct 30 20:01:11.018637 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:11.018587    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc6179186d5c5fe99146518ebf8c18f6151db8b7a762b9337b7b0f4457971e23"} err="failed to get container status \"fc6179186d5c5fe99146518ebf8c18f6151db8b7a762b9337b7b0f4457971e23\": rpc error: code = NotFound desc = could not find container \"fc6179186d5c5fe99146518ebf8c18f6151db8b7a762b9337b7b0f4457971e23\": container with ID starting with fc6179186d5c5fe99146518ebf8c18f6151db8b7a762b9337b7b0f4457971e23 not found: ID does not exist"
Oct 30 20:01:11.033923 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:11.033887    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"]
Oct 30 20:01:11.050702 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:11.050670    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-eaa5cd-openshift-default-d79bddd6c-sv85x"]
Oct 30 20:01:12.281323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:12.281282    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d703686a-ccb8-4f4a-8eac-15ce8d774149" path="/var/lib/kubelet/pods/d703686a-ccb8-4f4a-8eac-15ce8d774149/volumes"
Oct 30 20:01:13.717053 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.717015    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"]
Oct 30 20:01:13.717957 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.717937    4823 cpu_manager.go:414] "containerMap: removing container" podUID="c5f003b6-482e-4099-94af-1853828bdd2f" containerName="revoke-keys"
Oct 30 20:01:13.717957 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.717956    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f003b6-482e-4099-94af-1853828bdd2f" containerName="revoke-keys"
Oct 30 20:01:13.718127 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.717993    4823 cpu_manager.go:414] "containerMap: removing container" podUID="a1d1a2c2-5642-4e1d-9de9-edc32176a1f8" containerName="revoke-keys"
Oct 30 20:01:13.718127 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.717999    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d1a2c2-5642-4e1d-9de9-edc32176a1f8" containerName="revoke-keys"
Oct 30 20:01:13.718127 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.718019    4823 cpu_manager.go:414] "containerMap: removing container" podUID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerName="istio-proxy"
Oct 30 20:01:13.718127 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.718026    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerName="istio-proxy"
Oct 30 20:01:13.718261 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.718202    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f003b6-482e-4099-94af-1853828bdd2f" containerName="revoke-keys"
Oct 30 20:01:13.718261 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.718222    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d1a2c2-5642-4e1d-9de9-edc32176a1f8" containerName="revoke-keys"
Oct 30 20:01:13.718261 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.718231    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerName="istio-proxy"
Oct 30 20:01:13.718261 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.718240    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerName="istio-proxy"
Oct 30 20:01:13.722451 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.722425    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.749925 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.749890    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"]
Oct 30 20:01:13.751211 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.751169    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-workload-socket\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.751360 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.751315    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-data\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.751442 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.751421    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ad66e7-5505-4c0b-96f8-833adf2bd769-istiod-ca-cert\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.751582 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.751504    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-podinfo\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.751639 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.751609    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-token\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.751705 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.751642    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-credential-socket\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.751778 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.751760    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-workload-certs\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.751834 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.751821    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-envoy\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.752044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.752022    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqdxf\" (UniqueName: \"kubernetes.io/projected/35ad66e7-5505-4c0b-96f8-833adf2bd769-kube-api-access-gqdxf\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.753084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.753062    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-96ec93-openshift-default-dockercfg-g9m2l\""
Oct 30 20:01:13.941887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.918954    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-workload-socket\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.941887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.919082    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-data\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.941887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.919179    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ad66e7-5505-4c0b-96f8-833adf2bd769-istiod-ca-cert\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.941887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.919217    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-podinfo\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.941887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.919317    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-token\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.941887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.919367    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-credential-socket\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.941887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.919470    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-workload-certs\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.941887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.919520    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-envoy\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.941887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.919622    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqdxf\" (UniqueName: \"kubernetes.io/projected/35ad66e7-5505-4c0b-96f8-833adf2bd769-kube-api-access-gqdxf\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.941887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.933927    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-data\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.970935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.957276    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-credential-socket\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.970935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.958149    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-workload-socket\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.970935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.958384    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-workload-certs\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:13.970935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:13.959021    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ad66e7-5505-4c0b-96f8-833adf2bd769-istiod-ca-cert\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:14.063047 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:14.042490    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-token\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:14.142226 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:14.124548    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-envoy\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:14.142226 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:14.141577    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqdxf\" (UniqueName: \"kubernetes.io/projected/35ad66e7-5505-4c0b-96f8-833adf2bd769-kube-api-access-gqdxf\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:14.272131 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:14.272038    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-podinfo\") pod \"e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") " pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:14.339844 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:14.339541    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:14.732434 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:01:14.732403    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ad66e7_5505_4c0b_96f8_833adf2bd769.slice/crio-0dab86b4b1b3f2efd99a485e51eb0998c594a56abc26367b2cb04781701d6efb WatchSource:0}: Error finding container 0dab86b4b1b3f2efd99a485e51eb0998c594a56abc26367b2cb04781701d6efb: Status 404 returned error can't find the container with id 0dab86b4b1b3f2efd99a485e51eb0998c594a56abc26367b2cb04781701d6efb
Oct 30 20:01:14.732954 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:14.732559    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"]
Oct 30 20:01:14.735768 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:14.735736    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:01:14.735843 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:14.735805    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:01:14.735843 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:14.735836    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:01:15.350450 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:15.350405    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9" event={"ID":"35ad66e7-5505-4c0b-96f8-833adf2bd769","Type":"ContainerStarted","Data":"dccc357281ebafd063b095e06fb81e77a6fc66b00a9cd7a558b5ab49f7f09eb9"}
Oct 30 20:01:15.350631 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:15.350477    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9" event={"ID":"35ad66e7-5505-4c0b-96f8-833adf2bd769","Type":"ContainerStarted","Data":"0dab86b4b1b3f2efd99a485e51eb0998c594a56abc26367b2cb04781701d6efb"}
Oct 30 20:01:15.383878 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:15.383806    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9" podStartSLOduration=2.383792553 podStartE2EDuration="2.383792553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:15.38140132 +0000 UTC m=+1875.274015644" watchObservedRunningTime="2026-10-30 20:01:15.383792553 +0000 UTC m=+1875.276406996"
Oct 30 20:01:16.341299 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:16.341259    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:16.347723 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:16.347698    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:16.359724 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:16.359675    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:16.360765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:16.360739    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:01:18.376508 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.376474    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"]
Oct 30 20:01:18.377335 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.377318    4823 cpu_manager.go:414] "containerMap: removing container" podUID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerName="istio-proxy"
Oct 30 20:01:18.377388 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.377336    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d703686a-ccb8-4f4a-8eac-15ce8d774149" containerName="istio-proxy"
Oct 30 20:01:18.380660 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.380635    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"
Oct 30 20:01:18.394292 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.394267    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:18.394439 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.394347    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:01:18.402351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.402317    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"]
Oct 30 20:01:18.476008 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.475967    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09dc3997-433c-412b-94f9-a08a8663b763-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb\" (UID: \"09dc3997-433c-412b-94f9-a08a8663b763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"
Oct 30 20:01:18.476008 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.476016    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjzbv\" (UniqueName: \"kubernetes.io/projected/09dc3997-433c-412b-94f9-a08a8663b763-kube-api-access-qjzbv\") pod \"maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb\" (UID: \"09dc3997-433c-412b-94f9-a08a8663b763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"
Oct 30 20:01:18.476411 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.476391    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09dc3997-433c-412b-94f9-a08a8663b763-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb\" (UID: \"09dc3997-433c-412b-94f9-a08a8663b763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"
Oct 30 20:01:18.578664 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.578618    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09dc3997-433c-412b-94f9-a08a8663b763-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb\" (UID: \"09dc3997-433c-412b-94f9-a08a8663b763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"
Oct 30 20:01:18.579288 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.579258    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09dc3997-433c-412b-94f9-a08a8663b763-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb\" (UID: \"09dc3997-433c-412b-94f9-a08a8663b763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"
Oct 30 20:01:18.579490 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.579462    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzbv\" (UniqueName: \"kubernetes.io/projected/09dc3997-433c-412b-94f9-a08a8663b763-kube-api-access-qjzbv\") pod \"maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb\" (UID: \"09dc3997-433c-412b-94f9-a08a8663b763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"
Oct 30 20:01:18.581148 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.581126    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09dc3997-433c-412b-94f9-a08a8663b763-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb\" (UID: \"09dc3997-433c-412b-94f9-a08a8663b763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"
Oct 30 20:01:18.611802 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.611211    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09dc3997-433c-412b-94f9-a08a8663b763-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb\" (UID: \"09dc3997-433c-412b-94f9-a08a8663b763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"
Oct 30 20:01:18.655940 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.645114    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzbv\" (UniqueName: \"kubernetes.io/projected/09dc3997-433c-412b-94f9-a08a8663b763-kube-api-access-qjzbv\") pod \"maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb\" (UID: \"09dc3997-433c-412b-94f9-a08a8663b763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"
Oct 30 20:01:18.699198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:18.699151    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"
Oct 30 20:01:19.099993 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:19.099907    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"]
Oct 30 20:01:19.127258 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:19.126315    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"
Oct 30 20:01:19.141615 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:19.141551    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"]
Oct 30 20:01:19.329976 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:19.329941    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6chvf\" (UniqueName: \"kubernetes.io/projected/e82b58ca-46a9-49a4-84a8-38d75eb1048b-kube-api-access-6chvf\") pod \"maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf\" (UID: \"e82b58ca-46a9-49a4-84a8-38d75eb1048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"
Oct 30 20:01:19.330158 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:19.330081    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"]
Oct 30 20:01:19.330499 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:19.330470    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82b58ca-46a9-49a4-84a8-38d75eb1048b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf\" (UID: \"e82b58ca-46a9-49a4-84a8-38d75eb1048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"
Oct 30 20:01:19.330614 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:19.330513    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82b58ca-46a9-49a4-84a8-38d75eb1048b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf\" (UID: \"e82b58ca-46a9-49a4-84a8-38d75eb1048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"
Oct 30 20:01:19.330839 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:01:19.330811    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09dc3997_433c_412b_94f9_a08a8663b763.slice/crio-af18de202145e7c82fa920645a7b0542f1deb2fe2428c5658f7c15e01b431277 WatchSource:0}: Error finding container af18de202145e7c82fa920645a7b0542f1deb2fe2428c5658f7c15e01b431277: Status 404 returned error can't find the container with id af18de202145e7c82fa920645a7b0542f1deb2fe2428c5658f7c15e01b431277
Oct 30 20:01:19.393899 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:19.393821    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb" event={"ID":"09dc3997-433c-412b-94f9-a08a8663b763","Type":"ContainerStarted","Data":"af18de202145e7c82fa920645a7b0542f1deb2fe2428c5658f7c15e01b431277"}
Oct 30 20:01:19.432121 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:19.432083    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82b58ca-46a9-49a4-84a8-38d75eb1048b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf\" (UID: \"e82b58ca-46a9-49a4-84a8-38d75eb1048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"
Oct 30 20:01:19.432340 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:19.432134    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82b58ca-46a9-49a4-84a8-38d75eb1048b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf\" (UID: \"e82b58ca-46a9-49a4-84a8-38d75eb1048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"
Oct 30 20:01:19.432340 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:19.432295    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6chvf\" (UniqueName: \"kubernetes.io/projected/e82b58ca-46a9-49a4-84a8-38d75eb1048b-kube-api-access-6chvf\") pod \"maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf\" (UID: \"e82b58ca-46a9-49a4-84a8-38d75eb1048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"
Oct 30 20:01:19.432875 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:19.432831    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82b58ca-46a9-49a4-84a8-38d75eb1048b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf\" (UID: \"e82b58ca-46a9-49a4-84a8-38d75eb1048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"
Oct 30 20:01:19.434754 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:19.434734    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82b58ca-46a9-49a4-84a8-38d75eb1048b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf\" (UID: \"e82b58ca-46a9-49a4-84a8-38d75eb1048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"
Oct 30 20:01:19.445057 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:19.445029    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6chvf\" (UniqueName: \"kubernetes.io/projected/e82b58ca-46a9-49a4-84a8-38d75eb1048b-kube-api-access-6chvf\") pod \"maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf\" (UID: \"e82b58ca-46a9-49a4-84a8-38d75eb1048b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"
Oct 30 20:01:19.455053 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:19.455028    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"
Oct 30 20:01:20.293058 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:20.293023    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"]
Oct 30 20:01:20.471380 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:20.471335    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb" event={"ID":"09dc3997-433c-412b-94f9-a08a8663b763","Type":"ContainerStarted","Data":"0d7854199d14b38c4a13b310ac64460c57099c1ec2b413754ca5dacbb4964e4a"}
Oct 30 20:01:20.522470 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:20.522420    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf" event={"ID":"e82b58ca-46a9-49a4-84a8-38d75eb1048b","Type":"ContainerStarted","Data":"5101094cc9b38fe03f2512cb9be56fa5e6f982ba662b7b8f9b3dff5c8e6266dd"}
Oct 30 20:01:20.806411 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:20.806160    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb" podStartSLOduration=2.806137569 podStartE2EDuration="2.806137569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:20.523138059 +0000 UTC m=+1880.415752384" watchObservedRunningTime="2026-10-30 20:01:20.806137569 +0000 UTC m=+1880.698751893"
Oct 30 20:01:20.947945 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:20.922906    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"]
Oct 30 20:01:20.964192 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:20.957787    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-9r45j"]
Oct 30 20:01:21.538489 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:21.538447    4823 generic.go:356] "Generic (PLEG): container finished" podID="e82b58ca-46a9-49a4-84a8-38d75eb1048b" containerID="35070bc8e31926d17d8fa3a91bad19c1aa3b690d9920db8ce69e0076af573032" exitCode=0
Oct 30 20:01:21.539015 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:21.538522    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf" event={"ID":"e82b58ca-46a9-49a4-84a8-38d75eb1048b","Type":"ContainerDied","Data":"35070bc8e31926d17d8fa3a91bad19c1aa3b690d9920db8ce69e0076af573032"}
Oct 30 20:01:21.541008 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:21.540973    4823 generic.go:356] "Generic (PLEG): container finished" podID="09dc3997-433c-412b-94f9-a08a8663b763" containerID="0d7854199d14b38c4a13b310ac64460c57099c1ec2b413754ca5dacbb4964e4a" exitCode=0
Oct 30 20:01:21.541123 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:21.541051    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb" event={"ID":"09dc3997-433c-412b-94f9-a08a8663b763","Type":"ContainerDied","Data":"0d7854199d14b38c4a13b310ac64460c57099c1ec2b413754ca5dacbb4964e4a"}
Oct 30 20:01:21.644561 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:01:21.644515    4823 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82b58ca_46a9_49a4_84a8_38d75eb1048b.slice/crio-conmon-35070bc8e31926d17d8fa3a91bad19c1aa3b690d9920db8ce69e0076af573032.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:01:22.333267 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.333149    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1d1a2c2-5642-4e1d-9de9-edc32176a1f8" path="/var/lib/kubelet/pods/a1d1a2c2-5642-4e1d-9de9-edc32176a1f8/volumes"
Oct 30 20:01:22.754766 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.754735    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"
Oct 30 20:01:22.815133 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.815103    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"
Oct 30 20:01:22.848425 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.848390    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e82b58ca-46a9-49a4-84a8-38d75eb1048b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82b58ca-46a9-49a4-84a8-38d75eb1048b-maas-api-xks-ca\") pod \"e82b58ca-46a9-49a4-84a8-38d75eb1048b\" (UID: \"e82b58ca-46a9-49a4-84a8-38d75eb1048b\") "
Oct 30 20:01:22.848594 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.848523    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e82b58ca-46a9-49a4-84a8-38d75eb1048b-kube-api-access-6chvf\" (UniqueName: \"kubernetes.io/projected/e82b58ca-46a9-49a4-84a8-38d75eb1048b-kube-api-access-6chvf\") pod \"e82b58ca-46a9-49a4-84a8-38d75eb1048b\" (UID: \"e82b58ca-46a9-49a4-84a8-38d75eb1048b\") "
Oct 30 20:01:22.848745 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.848621    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e82b58ca-46a9-49a4-84a8-38d75eb1048b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82b58ca-46a9-49a4-84a8-38d75eb1048b-maas-api-openshift-ca\") pod \"e82b58ca-46a9-49a4-84a8-38d75eb1048b\" (UID: \"e82b58ca-46a9-49a4-84a8-38d75eb1048b\") "
Oct 30 20:01:22.848969 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.848946    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82b58ca-46a9-49a4-84a8-38d75eb1048b-maas-api-openshift-ca" pod "e82b58ca-46a9-49a4-84a8-38d75eb1048b" (UID: "e82b58ca-46a9-49a4-84a8-38d75eb1048b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:22.849502 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.849471    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82b58ca-46a9-49a4-84a8-38d75eb1048b-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:22.850775 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.850743    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82b58ca-46a9-49a4-84a8-38d75eb1048b-kube-api-access-6chvf" pod "e82b58ca-46a9-49a4-84a8-38d75eb1048b" (UID: "e82b58ca-46a9-49a4-84a8-38d75eb1048b"). InnerVolumeSpecName "kube-api-access-6chvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:22.850921 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.850806    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e82b58ca-46a9-49a4-84a8-38d75eb1048b-maas-api-xks-ca" pod "e82b58ca-46a9-49a4-84a8-38d75eb1048b" (UID: "e82b58ca-46a9-49a4-84a8-38d75eb1048b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:22.950370 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.950285    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09dc3997-433c-412b-94f9-a08a8663b763-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09dc3997-433c-412b-94f9-a08a8663b763-maas-api-openshift-ca\") pod \"09dc3997-433c-412b-94f9-a08a8663b763\" (UID: \"09dc3997-433c-412b-94f9-a08a8663b763\") "
Oct 30 20:01:22.950529 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.950423    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09dc3997-433c-412b-94f9-a08a8663b763-kube-api-access-qjzbv\" (UniqueName: \"kubernetes.io/projected/09dc3997-433c-412b-94f9-a08a8663b763-kube-api-access-qjzbv\") pod \"09dc3997-433c-412b-94f9-a08a8663b763\" (UID: \"09dc3997-433c-412b-94f9-a08a8663b763\") "
Oct 30 20:01:22.950648 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.950633    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09dc3997-433c-412b-94f9-a08a8663b763-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09dc3997-433c-412b-94f9-a08a8663b763-maas-api-xks-ca\") pod \"09dc3997-433c-412b-94f9-a08a8663b763\" (UID: \"09dc3997-433c-412b-94f9-a08a8663b763\") "
Oct 30 20:01:22.950689 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.950650    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09dc3997-433c-412b-94f9-a08a8663b763-maas-api-openshift-ca" pod "09dc3997-433c-412b-94f9-a08a8663b763" (UID: "09dc3997-433c-412b-94f9-a08a8663b763"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:22.951400 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.951374    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82b58ca-46a9-49a4-84a8-38d75eb1048b-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:22.951400 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.951397    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6chvf\" (UniqueName: \"kubernetes.io/projected/e82b58ca-46a9-49a4-84a8-38d75eb1048b-kube-api-access-6chvf\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:22.951603 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.951407    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09dc3997-433c-412b-94f9-a08a8663b763-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:22.952718 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.952684    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09dc3997-433c-412b-94f9-a08a8663b763-kube-api-access-qjzbv" pod "09dc3997-433c-412b-94f9-a08a8663b763" (UID: "09dc3997-433c-412b-94f9-a08a8663b763"). InnerVolumeSpecName "kube-api-access-qjzbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:22.952718 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:22.952696    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09dc3997-433c-412b-94f9-a08a8663b763-maas-api-xks-ca" pod "09dc3997-433c-412b-94f9-a08a8663b763" (UID: "09dc3997-433c-412b-94f9-a08a8663b763"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:23.053768 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:23.053733    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjzbv\" (UniqueName: \"kubernetes.io/projected/09dc3997-433c-412b-94f9-a08a8663b763-kube-api-access-qjzbv\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:23.053768 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:23.053765    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09dc3997-433c-412b-94f9-a08a8663b763-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:23.614726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:23.609155    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf" event={"ID":"e82b58ca-46a9-49a4-84a8-38d75eb1048b","Type":"ContainerDied","Data":"5101094cc9b38fe03f2512cb9be56fa5e6f982ba662b7b8f9b3dff5c8e6266dd"}
Oct 30 20:01:23.614726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:23.609239    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5101094cc9b38fe03f2512cb9be56fa5e6f982ba662b7b8f9b3dff5c8e6266dd"
Oct 30 20:01:23.614726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:23.609314    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"
Oct 30 20:01:23.646900 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:23.643366    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb" event={"ID":"09dc3997-433c-412b-94f9-a08a8663b763","Type":"ContainerDied","Data":"af18de202145e7c82fa920645a7b0542f1deb2fe2428c5658f7c15e01b431277"}
Oct 30 20:01:23.646900 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:23.643437    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af18de202145e7c82fa920645a7b0542f1deb2fe2428c5658f7c15e01b431277"
Oct 30 20:01:23.646900 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:23.643508    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"
Oct 30 20:01:26.547457 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:26.547402    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:26.635890 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:26.611083    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:26.635890 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:26.627450    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:26.646285 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:26.642075    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:26.756892 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:26.755139    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:27.046064 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:27.046005    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:27.151081 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:27.149998    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:27.343941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:27.329923    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"]
Oct 30 20:01:27.376438 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:27.376408    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"]
Oct 30 20:01:27.376853 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:27.376820    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc" podUID="9d0dea3a-53d6-425f-a202-506d89080874" containerName="istio-proxy" containerID="cri-o://ce6cc0cac78963b2a50d327839b01867b4c3c47af28467b06d55c5e251b38d31" gracePeriod=30
Oct 30 20:01:27.402888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:27.377360    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps" podUID="181fb3fa-5478-4c39-9cfe-de04f2070e49" containerName="istio-proxy" containerID="cri-o://502e0d62e64e8d721ab9505ac6578863ee10ee0c86be3c02f492c2df29d9b53c" gracePeriod=30
Oct 30 20:01:27.490109 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:27.436259    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:27.759361 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:27.754007    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 20:01:27.759361 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:27.754436    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 20:01:27.759361 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:27.754773    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 20:01:27.759361 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:27.754833    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 20:01:27.759361 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:27.758640    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:27.759361 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:27.758996    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 20:01:27.773886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:27.772141    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:27.773886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:27.772567    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 20:01:27.855886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:27.846840    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:28.173281 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.059063    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:28.173281 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.079301    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 20:01:28.173281 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.168532    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:28.173281 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.170508    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 20:01:28.209897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.189952    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:28.210229 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.210189    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:28.459887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.442682    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:28.754939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.742536    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 20:01:28.756049 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.756011    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:28.785653 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.757929    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:28.790883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.762402    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:28.816529 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.795143    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:28.840747 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.819520    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 20:01:28.865825 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.847443    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:28.865825 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.848718    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 20:01:28.865825 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.860487    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Oct 30 20:01:28.931424 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.931279    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw"
Oct 30 20:01:28.999721 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:28.984664    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 20:01:29.254947 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:29.059385    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:29.417886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:29.406991    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 20:01:29.473892 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:29.472981    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 30 20:01:29.629207 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:29.629177    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 20:01:29.692603 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:29.692573    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 20:01:29.710509 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:29.701155    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 20:01:29.731700 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:29.725053    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 20:01:32.029222 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:01:32.029189    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:01:32.849317 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:32.849281    4823 generic.go:356] "Generic (PLEG): container finished" podID="9d0dea3a-53d6-425f-a202-506d89080874" containerID="ce6cc0cac78963b2a50d327839b01867b4c3c47af28467b06d55c5e251b38d31" exitCode=0
Oct 30 20:01:32.849941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:32.849443    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc" event={"ID":"9d0dea3a-53d6-425f-a202-506d89080874","Type":"ContainerDied","Data":"ce6cc0cac78963b2a50d327839b01867b4c3c47af28467b06d55c5e251b38d31"}
Oct 30 20:01:32.923668 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:32.923633    4823 generic.go:356] "Generic (PLEG): container finished" podID="181fb3fa-5478-4c39-9cfe-de04f2070e49" containerID="502e0d62e64e8d721ab9505ac6578863ee10ee0c86be3c02f492c2df29d9b53c" exitCode=0
Oct 30 20:01:32.923855 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:32.923718    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps" event={"ID":"181fb3fa-5478-4c39-9cfe-de04f2070e49","Type":"ContainerDied","Data":"502e0d62e64e8d721ab9505ac6578863ee10ee0c86be3c02f492c2df29d9b53c"}
Oct 30 20:01:33.349276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.349247    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 20:01:33.441997 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.441965    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 20:01:33.536200 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.536165    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-workload-socket\") pod \"9d0dea3a-53d6-425f-a202-506d89080874\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") "
Oct 30 20:01:33.536357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.536281    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-workload-certs\") pod \"9d0dea3a-53d6-425f-a202-506d89080874\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") "
Oct 30 20:01:33.536357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.536306    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-credential-socket\") pod \"9d0dea3a-53d6-425f-a202-506d89080874\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") "
Oct 30 20:01:33.536357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.536339    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0dea3a-53d6-425f-a202-506d89080874-kube-api-access-lrw4b\" (UniqueName: \"kubernetes.io/projected/9d0dea3a-53d6-425f-a202-506d89080874-kube-api-access-lrw4b\") pod \"9d0dea3a-53d6-425f-a202-506d89080874\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") "
Oct 30 20:01:33.536468 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.536371    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d0dea3a-53d6-425f-a202-506d89080874-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0dea3a-53d6-425f-a202-506d89080874-istio-podinfo\") pod \"9d0dea3a-53d6-425f-a202-506d89080874\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") "
Oct 30 20:01:33.536468 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.536437    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-istio-data\") pod \"9d0dea3a-53d6-425f-a202-506d89080874\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") "
Oct 30 20:01:33.536577 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.536563    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0dea3a-53d6-425f-a202-506d89080874-istio-token\" (UniqueName: \"kubernetes.io/projected/9d0dea3a-53d6-425f-a202-506d89080874-istio-token\") pod \"9d0dea3a-53d6-425f-a202-506d89080874\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") "
Oct 30 20:01:33.536717 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.536692    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-credential-socket" pod "9d0dea3a-53d6-425f-a202-506d89080874" (UID: "9d0dea3a-53d6-425f-a202-506d89080874"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:33.536788 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.536776    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0dea3a-53d6-425f-a202-506d89080874-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0dea3a-53d6-425f-a202-506d89080874-istiod-ca-cert\") pod \"9d0dea3a-53d6-425f-a202-506d89080874\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") "
Oct 30 20:01:33.536852 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.536844    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-istio-envoy\") pod \"9d0dea3a-53d6-425f-a202-506d89080874\" (UID: \"9d0dea3a-53d6-425f-a202-506d89080874\") "
Oct 30 20:01:33.537628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.536846    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-workload-socket" pod "9d0dea3a-53d6-425f-a202-506d89080874" (UID: "9d0dea3a-53d6-425f-a202-506d89080874"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:33.537726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.537005    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-workload-certs" pod "9d0dea3a-53d6-425f-a202-506d89080874" (UID: "9d0dea3a-53d6-425f-a202-506d89080874"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:33.537726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.537352    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0dea3a-53d6-425f-a202-506d89080874-istiod-ca-cert" pod "9d0dea3a-53d6-425f-a202-506d89080874" (UID: "9d0dea3a-53d6-425f-a202-506d89080874"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:33.537726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.537594    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-istio-data" pod "9d0dea3a-53d6-425f-a202-506d89080874" (UID: "9d0dea3a-53d6-425f-a202-506d89080874"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:33.538003 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.537982    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.538084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.538012    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.538084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.538027    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.538084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.538041    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0dea3a-53d6-425f-a202-506d89080874-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.538084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.538054    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.539966 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.539938    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0dea3a-53d6-425f-a202-506d89080874-kube-api-access-lrw4b" pod "9d0dea3a-53d6-425f-a202-506d89080874" (UID: "9d0dea3a-53d6-425f-a202-506d89080874"). InnerVolumeSpecName "kube-api-access-lrw4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:33.540392 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.540367    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d0dea3a-53d6-425f-a202-506d89080874-istio-podinfo" pod "9d0dea3a-53d6-425f-a202-506d89080874" (UID: "9d0dea3a-53d6-425f-a202-506d89080874"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:33.540566 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.540537    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-istio-envoy" pod "9d0dea3a-53d6-425f-a202-506d89080874" (UID: "9d0dea3a-53d6-425f-a202-506d89080874"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:33.540835 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.540813    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0dea3a-53d6-425f-a202-506d89080874-istio-token" pod "9d0dea3a-53d6-425f-a202-506d89080874" (UID: "9d0dea3a-53d6-425f-a202-506d89080874"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:33.641247 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.641120    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-workload-certs\") pod \"181fb3fa-5478-4c39-9cfe-de04f2070e49\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") "
Oct 30 20:01:33.643806 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.643770    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-data\") pod \"181fb3fa-5478-4c39-9cfe-de04f2070e49\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") "
Oct 30 20:01:33.644023 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.643912    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-workload-socket\") pod \"181fb3fa-5478-4c39-9cfe-de04f2070e49\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") "
Oct 30 20:01:33.644023 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.643979    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181fb3fa-5478-4c39-9cfe-de04f2070e49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181fb3fa-5478-4c39-9cfe-de04f2070e49-istiod-ca-cert\") pod \"181fb3fa-5478-4c39-9cfe-de04f2070e49\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") "
Oct 30 20:01:33.644150 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.644118    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-token\" (UniqueName: \"kubernetes.io/projected/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-token\") pod \"181fb3fa-5478-4c39-9cfe-de04f2070e49\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") "
Oct 30 20:01:33.644218 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.644193    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181fb3fa-5478-4c39-9cfe-de04f2070e49-kube-api-access-t9tq8\" (UniqueName: \"kubernetes.io/projected/181fb3fa-5478-4c39-9cfe-de04f2070e49-kube-api-access-t9tq8\") pod \"181fb3fa-5478-4c39-9cfe-de04f2070e49\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") "
Oct 30 20:01:33.644279 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.644254    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-credential-socket\") pod \"181fb3fa-5478-4c39-9cfe-de04f2070e49\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") "
Oct 30 20:01:33.644397 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.644374    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-envoy\") pod \"181fb3fa-5478-4c39-9cfe-de04f2070e49\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") "
Oct 30 20:01:33.644592 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.644569    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-podinfo\") pod \"181fb3fa-5478-4c39-9cfe-de04f2070e49\" (UID: \"181fb3fa-5478-4c39-9cfe-de04f2070e49\") "
Oct 30 20:01:33.645844 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.645813    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-credential-socket" pod "181fb3fa-5478-4c39-9cfe-de04f2070e49" (UID: "181fb3fa-5478-4c39-9cfe-de04f2070e49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:33.662897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.661985    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-workload-certs" pod "181fb3fa-5478-4c39-9cfe-de04f2070e49" (UID: "181fb3fa-5478-4c39-9cfe-de04f2070e49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:33.662897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.662840    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0dea3a-53d6-425f-a202-506d89080874-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.662897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.662886    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0dea3a-53d6-425f-a202-506d89080874-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.662897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.662902    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.663383 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.662919    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrw4b\" (UniqueName: \"kubernetes.io/projected/9d0dea3a-53d6-425f-a202-506d89080874-kube-api-access-lrw4b\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.663383 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.662934    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0dea3a-53d6-425f-a202-506d89080874-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.663383 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.662948    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.667887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.664436    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-token" pod "181fb3fa-5478-4c39-9cfe-de04f2070e49" (UID: "181fb3fa-5478-4c39-9cfe-de04f2070e49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:33.667887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.664808    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181fb3fa-5478-4c39-9cfe-de04f2070e49-istiod-ca-cert" pod "181fb3fa-5478-4c39-9cfe-de04f2070e49" (UID: "181fb3fa-5478-4c39-9cfe-de04f2070e49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:33.673230 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.673166    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181fb3fa-5478-4c39-9cfe-de04f2070e49-kube-api-access-t9tq8" pod "181fb3fa-5478-4c39-9cfe-de04f2070e49" (UID: "181fb3fa-5478-4c39-9cfe-de04f2070e49"). InnerVolumeSpecName "kube-api-access-t9tq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:33.692740 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.679973    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-workload-socket" pod "181fb3fa-5478-4c39-9cfe-de04f2070e49" (UID: "181fb3fa-5478-4c39-9cfe-de04f2070e49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:33.692740 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.680083    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-data" pod "181fb3fa-5478-4c39-9cfe-de04f2070e49" (UID: "181fb3fa-5478-4c39-9cfe-de04f2070e49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:33.713485 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.713354    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-podinfo" pod "181fb3fa-5478-4c39-9cfe-de04f2070e49" (UID: "181fb3fa-5478-4c39-9cfe-de04f2070e49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:33.713986 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.713941    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-envoy" pod "181fb3fa-5478-4c39-9cfe-de04f2070e49" (UID: "181fb3fa-5478-4c39-9cfe-de04f2070e49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:33.772903 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.765709    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181fb3fa-5478-4c39-9cfe-de04f2070e49-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.772903 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.765751    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.772903 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.765769    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9tq8\" (UniqueName: \"kubernetes.io/projected/181fb3fa-5478-4c39-9cfe-de04f2070e49-kube-api-access-t9tq8\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.772903 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.765785    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.772903 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.765799    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.772903 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.765812    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.772903 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.765824    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181fb3fa-5478-4c39-9cfe-de04f2070e49-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:33.936920 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.936818    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"
Oct 30 20:01:33.936920 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.936846    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc" event={"ID":"9d0dea3a-53d6-425f-a202-506d89080874","Type":"ContainerDied","Data":"68afaa5a6f8a40b22e796382b80a2a373a8b873241f01939cdc9a041374fe5e7"}
Oct 30 20:01:33.936920 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.936917    4823 scope.go:122] "RemoveContainer" containerID="ce6cc0cac78963b2a50d327839b01867b4c3c47af28467b06d55c5e251b38d31"
Oct 30 20:01:33.939950 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.939929    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps" event={"ID":"181fb3fa-5478-4c39-9cfe-de04f2070e49","Type":"ContainerDied","Data":"9d8edc83d6a64380218b7bb3742c49815f85759d51c263a92ef572513a2c55bc"}
Oct 30 20:01:33.940110 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:33.940000    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"
Oct 30 20:01:34.045646 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:34.037314    4823 scope.go:122] "RemoveContainer" containerID="502e0d62e64e8d721ab9505ac6578863ee10ee0c86be3c02f492c2df29d9b53c"
Oct 30 20:01:34.049881 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:34.049845    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"]
Oct 30 20:01:34.109047 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:34.109012    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-l6lps"]
Oct 30 20:01:34.117805 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:34.117771    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"]
Oct 30 20:01:34.124436 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:34.124319    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-vmhbc"]
Oct 30 20:01:34.292059 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:34.291955    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181fb3fa-5478-4c39-9cfe-de04f2070e49" path="/var/lib/kubelet/pods/181fb3fa-5478-4c39-9cfe-de04f2070e49/volumes"
Oct 30 20:01:34.292611 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:34.292589    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0dea3a-53d6-425f-a202-506d89080874" path="/var/lib/kubelet/pods/9d0dea3a-53d6-425f-a202-506d89080874/volumes"
Oct 30 20:01:35.916014 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:35.915982    4823 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:01:35.916454 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:35.916032    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2" podUID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:39.468082 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:39.468037    4823 patch_prober.go:29] interesting pod/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:01:40.284413 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:39.468112    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx" podUID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:01:41.051012 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:41.050919    4823 generic.go:356] "Generic (PLEG): container finished" podID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerID="b8ed26a374750bd61410b7fcf446c3189fe44053e8a8cc56e1a1ea9565a260b0" exitCode=0
Oct 30 20:01:41.051622 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:41.051042    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2" event={"ID":"7b162ce0-cc0c-4f76-802d-9175c518f6df","Type":"ContainerDied","Data":"b8ed26a374750bd61410b7fcf446c3189fe44053e8a8cc56e1a1ea9565a260b0"}
Oct 30 20:01:41.051622 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:41.051603    4823 scope.go:122] "RemoveContainer" containerID="b8ed26a374750bd61410b7fcf446c3189fe44053e8a8cc56e1a1ea9565a260b0"
Oct 30 20:01:41.053162 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:41.053140    4823 generic.go:356] "Generic (PLEG): container finished" podID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerID="ea9a871b7f3e8a98eac8e369bc9ddb8e7687c6acde96b1d79b741f830fa82a6e" exitCode=0
Oct 30 20:01:41.053289 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:41.053214    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx" event={"ID":"f6660e61-7ad3-45cd-873e-d4c0a1741317","Type":"ContainerDied","Data":"ea9a871b7f3e8a98eac8e369bc9ddb8e7687c6acde96b1d79b741f830fa82a6e"}
Oct 30 20:01:41.053624 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:41.053609    4823 scope.go:122] "RemoveContainer" containerID="ea9a871b7f3e8a98eac8e369bc9ddb8e7687c6acde96b1d79b741f830fa82a6e"
Oct 30 20:01:41.053780 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:41.053749    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:01:41.053919 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:41.053825    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:01:41.053919 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:41.053857    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:01:41.055666 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:41.055628    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:01:41.055777 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:41.055714    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:01:41.055777 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:41.055771    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:01:41.065239 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:41.065182    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"]
Oct 30 20:01:41.074898 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:41.074804    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a81713d6-xjpj7"]
Oct 30 20:01:42.069964 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:42.069918    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx" event={"ID":"f6660e61-7ad3-45cd-873e-d4c0a1741317","Type":"ContainerStarted","Data":"522140d6bce10b9f4ff4110a959e67881f70a0979a0667457a2fa7168a6b112d"}
Oct 30 20:01:42.076189 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:42.076145    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2" event={"ID":"7b162ce0-cc0c-4f76-802d-9175c518f6df","Type":"ContainerStarted","Data":"e63b55df1ce8933b27855603072ce302aff50b293456e330fd14f583d347d982"}
Oct 30 20:01:42.328507 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:42.328409    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd2b7d3-be2d-446d-9824-6d89903b0938" path="/var/lib/kubelet/pods/afd2b7d3-be2d-446d-9824-6d89903b0938/volumes"
Oct 30 20:01:42.394231 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:42.372303    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:01:42.394231 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:42.372451    4823 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 20:01:42.394231 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:42.372500    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2" podUID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 20:01:42.394231 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:42.391368    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 20:01:42.443882 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:42.439390    4823 patch_prober.go:29] interesting pod/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 20:01:42.443882 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:42.439452    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx" podUID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 20:01:43.375887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:43.374218    4823 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 20:01:43.375887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:43.374299    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2" podUID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 20:01:43.391632 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:43.382202    4823 patch_prober.go:29] interesting pod/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 20:01:43.391632 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:43.382262    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx" podUID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 20:01:44.360885 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:44.360291    4823 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 20:01:44.360885 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:44.360348    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2" podUID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 20:01:44.398348 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:44.398312    4823 patch_prober.go:29] interesting pod/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 20:01:44.399104 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:44.399069    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx" podUID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 20:01:45.378886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:45.378296    4823 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 20:01:45.378886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:45.378364    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2" podUID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 20:01:45.400885 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:45.400429    4823 patch_prober.go:29] interesting pod/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 20:01:45.400885 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:45.400484    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx" podUID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 20:01:46.088541 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:46.088510    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"]
Oct 30 20:01:46.096884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:46.096838    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-98a0d8a9-b5wrb"]
Oct 30 20:01:46.315052 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:46.315013    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09dc3997-433c-412b-94f9-a08a8663b763" path="/var/lib/kubelet/pods/09dc3997-433c-412b-94f9-a08a8663b763/volumes"
Oct 30 20:01:46.357291 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:46.357255    4823 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 20:01:46.357597 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:46.357564    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2" podUID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 20:01:46.383744 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:46.383628    4823 patch_prober.go:29] interesting pod/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 20:01:46.384073 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:46.384040    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx" podUID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 20:01:47.373889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:47.370115    4823 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 20:01:47.373889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:47.370189    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2" podUID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 20:01:47.389910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:47.389853    4823 patch_prober.go:29] interesting pod/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 20:01:47.390283 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:47.390249    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx" podUID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 20:01:48.357573 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:48.357532    4823 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 20:01:48.357743 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:48.357584    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2" podUID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 20:01:48.395890 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:48.393222    4823 patch_prober.go:29] interesting pod/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 20:01:48.395890 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:48.393312    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx" podUID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 20:01:49.361915 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:49.361874    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:01:49.362388 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:49.362373    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:01:49.363221 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:49.363205    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:01:49.385545 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:49.385512    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 20:01:49.385941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:49.385898    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 20:01:49.387348 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:49.387320    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 20:01:53.737282 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.737244    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"]
Oct 30 20:01:53.738332 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.738307    4823 cpu_manager.go:414] "containerMap: removing container" podUID="e82b58ca-46a9-49a4-84a8-38d75eb1048b" containerName="revoke-keys"
Oct 30 20:01:53.738332 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.738332    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e82b58ca-46a9-49a4-84a8-38d75eb1048b" containerName="revoke-keys"
Oct 30 20:01:53.738512 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.738356    4823 cpu_manager.go:414] "containerMap: removing container" podUID="181fb3fa-5478-4c39-9cfe-de04f2070e49" containerName="istio-proxy"
Oct 30 20:01:53.738512 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.738363    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181fb3fa-5478-4c39-9cfe-de04f2070e49" containerName="istio-proxy"
Oct 30 20:01:53.738512 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.738404    4823 cpu_manager.go:414] "containerMap: removing container" podUID="09dc3997-433c-412b-94f9-a08a8663b763" containerName="revoke-keys"
Oct 30 20:01:53.738512 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.738412    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09dc3997-433c-412b-94f9-a08a8663b763" containerName="revoke-keys"
Oct 30 20:01:53.738512 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.738459    4823 cpu_manager.go:414] "containerMap: removing container" podUID="9d0dea3a-53d6-425f-a202-506d89080874" containerName="istio-proxy"
Oct 30 20:01:53.738512 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.738467    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0dea3a-53d6-425f-a202-506d89080874" containerName="istio-proxy"
Oct 30 20:01:53.738818 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.738670    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="09dc3997-433c-412b-94f9-a08a8663b763" containerName="revoke-keys"
Oct 30 20:01:53.738818 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.738691    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0dea3a-53d6-425f-a202-506d89080874" containerName="istio-proxy"
Oct 30 20:01:53.738818 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.738708    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="e82b58ca-46a9-49a4-84a8-38d75eb1048b" containerName="revoke-keys"
Oct 30 20:01:53.738818 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.738721    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="181fb3fa-5478-4c39-9cfe-de04f2070e49" containerName="istio-proxy"
Oct 30 20:01:53.742727 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.742705    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"
Oct 30 20:01:53.752706 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.752615    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:53.752706 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.752668    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:01:53.754698 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.754023    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"]
Oct 30 20:01:53.774598 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.774553    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrv66\" (UniqueName: \"kubernetes.io/projected/c7558ea4-5322-443f-9847-857688238559-kube-api-access-xrv66\") pod \"maas-api-revoke-keys-e2e-api-a-35deec-24nt4\" (UID: \"c7558ea4-5322-443f-9847-857688238559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"
Oct 30 20:01:53.774779 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.774627    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7558ea4-5322-443f-9847-857688238559-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-35deec-24nt4\" (UID: \"c7558ea4-5322-443f-9847-857688238559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"
Oct 30 20:01:53.774779 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.774738    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7558ea4-5322-443f-9847-857688238559-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-35deec-24nt4\" (UID: \"c7558ea4-5322-443f-9847-857688238559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"
Oct 30 20:01:53.876906 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.876853    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrv66\" (UniqueName: \"kubernetes.io/projected/c7558ea4-5322-443f-9847-857688238559-kube-api-access-xrv66\") pod \"maas-api-revoke-keys-e2e-api-a-35deec-24nt4\" (UID: \"c7558ea4-5322-443f-9847-857688238559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"
Oct 30 20:01:53.877116 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.876942    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7558ea4-5322-443f-9847-857688238559-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-35deec-24nt4\" (UID: \"c7558ea4-5322-443f-9847-857688238559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"
Oct 30 20:01:53.877116 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.876976    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7558ea4-5322-443f-9847-857688238559-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-35deec-24nt4\" (UID: \"c7558ea4-5322-443f-9847-857688238559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"
Oct 30 20:01:53.877741 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.877721    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7558ea4-5322-443f-9847-857688238559-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-35deec-24nt4\" (UID: \"c7558ea4-5322-443f-9847-857688238559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"
Oct 30 20:01:53.879560 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.879540    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7558ea4-5322-443f-9847-857688238559-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-35deec-24nt4\" (UID: \"c7558ea4-5322-443f-9847-857688238559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"
Oct 30 20:01:53.893478 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:53.893446    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrv66\" (UniqueName: \"kubernetes.io/projected/c7558ea4-5322-443f-9847-857688238559-kube-api-access-xrv66\") pod \"maas-api-revoke-keys-e2e-api-a-35deec-24nt4\" (UID: \"c7558ea4-5322-443f-9847-857688238559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"
Oct 30 20:01:54.060721 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:54.060644    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"
Oct 30 20:01:54.383163 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:54.383131    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"]
Oct 30 20:01:54.384655 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:01:54.384629    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7558ea4_5322_443f_9847_857688238559.slice/crio-2382309448f887057062694a3937876d9ee5184444ca20ee43ffe146ab77a610 WatchSource:0}: Error finding container 2382309448f887057062694a3937876d9ee5184444ca20ee43ffe146ab77a610: Status 404 returned error can't find the container with id 2382309448f887057062694a3937876d9ee5184444ca20ee43ffe146ab77a610
Oct 30 20:01:54.402922 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:54.402891    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4" event={"ID":"c7558ea4-5322-443f-9847-857688238559","Type":"ContainerStarted","Data":"2382309448f887057062694a3937876d9ee5184444ca20ee43ffe146ab77a610"}
Oct 30 20:01:55.470516 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:55.470431    4823 generic.go:356] "Generic (PLEG): container finished" podID="c7558ea4-5322-443f-9847-857688238559" containerID="88a31d81ec2fd051a7308e794d1792039203c7e4d32becc4513d7fbb31542899" exitCode=0
Oct 30 20:01:55.471334 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:55.470820    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4" event={"ID":"c7558ea4-5322-443f-9847-857688238559","Type":"ContainerDied","Data":"88a31d81ec2fd051a7308e794d1792039203c7e4d32becc4513d7fbb31542899"}
Oct 30 20:01:56.044264 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:56.044223    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"]
Oct 30 20:01:56.044531 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:56.044499    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx" podUID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerName="istio-proxy" containerID="cri-o://522140d6bce10b9f4ff4110a959e67881f70a0979a0667457a2fa7168a6b112d" gracePeriod=30
Oct 30 20:01:56.697159 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:56.697127    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"
Oct 30 20:01:56.870507 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:56.869626    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7558ea4-5322-443f-9847-857688238559-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7558ea4-5322-443f-9847-857688238559-maas-api-openshift-ca\") pod \"c7558ea4-5322-443f-9847-857688238559\" (UID: \"c7558ea4-5322-443f-9847-857688238559\") "
Oct 30 20:01:56.870507 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:56.869731    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7558ea4-5322-443f-9847-857688238559-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7558ea4-5322-443f-9847-857688238559-maas-api-xks-ca\") pod \"c7558ea4-5322-443f-9847-857688238559\" (UID: \"c7558ea4-5322-443f-9847-857688238559\") "
Oct 30 20:01:56.870507 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:56.870029    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7558ea4-5322-443f-9847-857688238559-kube-api-access-xrv66\" (UniqueName: \"kubernetes.io/projected/c7558ea4-5322-443f-9847-857688238559-kube-api-access-xrv66\") pod \"c7558ea4-5322-443f-9847-857688238559\" (UID: \"c7558ea4-5322-443f-9847-857688238559\") "
Oct 30 20:01:56.874026 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:56.872309    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7558ea4-5322-443f-9847-857688238559-maas-api-openshift-ca" pod "c7558ea4-5322-443f-9847-857688238559" (UID: "c7558ea4-5322-443f-9847-857688238559"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:56.879596 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:56.879567    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7558ea4-5322-443f-9847-857688238559-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:56.906261 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:56.906100    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7558ea4-5322-443f-9847-857688238559-maas-api-xks-ca" pod "c7558ea4-5322-443f-9847-857688238559" (UID: "c7558ea4-5322-443f-9847-857688238559"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:56.911682 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:56.911645    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7558ea4-5322-443f-9847-857688238559-kube-api-access-xrv66" pod "c7558ea4-5322-443f-9847-857688238559" (UID: "c7558ea4-5322-443f-9847-857688238559"). InnerVolumeSpecName "kube-api-access-xrv66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:56.981637 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:56.981604    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrv66\" (UniqueName: \"kubernetes.io/projected/c7558ea4-5322-443f-9847-857688238559-kube-api-access-xrv66\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:56.981637 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:56.981638    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7558ea4-5322-443f-9847-857688238559-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:57.665913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:57.659709    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4" event={"ID":"c7558ea4-5322-443f-9847-857688238559","Type":"ContainerDied","Data":"2382309448f887057062694a3937876d9ee5184444ca20ee43ffe146ab77a610"}
Oct 30 20:01:57.665913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:57.659777    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2382309448f887057062694a3937876d9ee5184444ca20ee43ffe146ab77a610"
Oct 30 20:01:57.665913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:01:57.659852    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"
Oct 30 20:02:01.705910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.705832    4823 generic.go:356] "Generic (PLEG): container finished" podID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerID="522140d6bce10b9f4ff4110a959e67881f70a0979a0667457a2fa7168a6b112d" exitCode=0
Oct 30 20:02:01.705910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.705892    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx" event={"ID":"f6660e61-7ad3-45cd-873e-d4c0a1741317","Type":"ContainerDied","Data":"522140d6bce10b9f4ff4110a959e67881f70a0979a0667457a2fa7168a6b112d"}
Oct 30 20:02:01.706427 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.705958    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx" event={"ID":"f6660e61-7ad3-45cd-873e-d4c0a1741317","Type":"ContainerDied","Data":"6f02dac36f813be1aa5f0a353dfdcfb48367f2bd927e77dede2f2e1810683885"}
Oct 30 20:02:01.706427 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.705982    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f02dac36f813be1aa5f0a353dfdcfb48367f2bd927e77dede2f2e1810683885"
Oct 30 20:02:01.706427 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.706000    4823 scope.go:122] "RemoveContainer" containerID="ea9a871b7f3e8a98eac8e369bc9ddb8e7687c6acde96b1d79b741f830fa82a6e"
Oct 30 20:02:01.707281 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.707257    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 20:02:01.773711 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.773680    4823 scope.go:122] "RemoveContainer" containerID="e7d783ece075943f726e56b83e53e126f2f6a843fcbd42627c1a646435bf8530"
Oct 30 20:02:01.783406 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.783378    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6660e61-7ad3-45cd-873e-d4c0a1741317-kube-api-access-dxpzx\" (UniqueName: \"kubernetes.io/projected/f6660e61-7ad3-45cd-873e-d4c0a1741317-kube-api-access-dxpzx\") pod \"f6660e61-7ad3-45cd-873e-d4c0a1741317\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") "
Oct 30 20:02:01.783545 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.783428    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-envoy\") pod \"f6660e61-7ad3-45cd-873e-d4c0a1741317\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") "
Oct 30 20:02:01.783662 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.783643    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6660e61-7ad3-45cd-873e-d4c0a1741317-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6660e61-7ad3-45cd-873e-d4c0a1741317-istiod-ca-cert\") pod \"f6660e61-7ad3-45cd-873e-d4c0a1741317\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") "
Oct 30 20:02:01.784060 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.783727    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-token\" (UniqueName: \"kubernetes.io/projected/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-token\") pod \"f6660e61-7ad3-45cd-873e-d4c0a1741317\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") "
Oct 30 20:02:01.784060 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.783803    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-podinfo\") pod \"f6660e61-7ad3-45cd-873e-d4c0a1741317\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") "
Oct 30 20:02:01.784060 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.783831    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-credential-socket\") pod \"f6660e61-7ad3-45cd-873e-d4c0a1741317\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") "
Oct 30 20:02:01.784060 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.783986    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-workload-socket\") pod \"f6660e61-7ad3-45cd-873e-d4c0a1741317\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") "
Oct 30 20:02:01.784358 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.784062    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-workload-certs\") pod \"f6660e61-7ad3-45cd-873e-d4c0a1741317\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") "
Oct 30 20:02:01.784358 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.784107    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-data\") pod \"f6660e61-7ad3-45cd-873e-d4c0a1741317\" (UID: \"f6660e61-7ad3-45cd-873e-d4c0a1741317\") "
Oct 30 20:02:01.784459 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.784391    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6660e61-7ad3-45cd-873e-d4c0a1741317-istiod-ca-cert" pod "f6660e61-7ad3-45cd-873e-d4c0a1741317" (UID: "f6660e61-7ad3-45cd-873e-d4c0a1741317"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:01.784658 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.784633    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-data" pod "f6660e61-7ad3-45cd-873e-d4c0a1741317" (UID: "f6660e61-7ad3-45cd-873e-d4c0a1741317"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:01.784929 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.784836    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-credential-socket" pod "f6660e61-7ad3-45cd-873e-d4c0a1741317" (UID: "f6660e61-7ad3-45cd-873e-d4c0a1741317"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:01.785088 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.785060    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-workload-socket" pod "f6660e61-7ad3-45cd-873e-d4c0a1741317" (UID: "f6660e61-7ad3-45cd-873e-d4c0a1741317"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:01.785320 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.785296    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-workload-certs" pod "f6660e61-7ad3-45cd-873e-d4c0a1741317" (UID: "f6660e61-7ad3-45cd-873e-d4c0a1741317"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:01.785396 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.785342    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6660e61-7ad3-45cd-873e-d4c0a1741317-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:01.785396 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.785367    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:01.785396 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.785381    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:01.785632 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.785397    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:01.786221 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.786193    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-envoy" pod "f6660e61-7ad3-45cd-873e-d4c0a1741317" (UID: "f6660e61-7ad3-45cd-873e-d4c0a1741317"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:01.786758 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.786741    4823 scope.go:122] "RemoveContainer" containerID="7347190d7c58d04089d4081a518ec6a76ec37dbc2031013025fff8f649fe55b6"
Oct 30 20:02:01.787242 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.787217    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6660e61-7ad3-45cd-873e-d4c0a1741317-kube-api-access-dxpzx" pod "f6660e61-7ad3-45cd-873e-d4c0a1741317" (UID: "f6660e61-7ad3-45cd-873e-d4c0a1741317"). InnerVolumeSpecName "kube-api-access-dxpzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:01.789130 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.789107    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-token" pod "f6660e61-7ad3-45cd-873e-d4c0a1741317" (UID: "f6660e61-7ad3-45cd-873e-d4c0a1741317"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:01.789213 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.789155    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-podinfo" pod "f6660e61-7ad3-45cd-873e-d4c0a1741317" (UID: "f6660e61-7ad3-45cd-873e-d4c0a1741317"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:01.827450 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.827419    4823 scope.go:122] "RemoveContainer" containerID="ecaf5cebb29e670414a2431477d240305f2f7ef5de48d2784b5836309be37aa6"
Oct 30 20:02:01.838462 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.838437    4823 scope.go:122] "RemoveContainer" containerID="ca570e1486cbc1b6c8b4201a51f41b9111346a63040830e0fd4a2f9709408569"
Oct 30 20:02:01.849450 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.849430    4823 scope.go:122] "RemoveContainer" containerID="bd93abdec51aedfe2c151cbf3f15fc1f85c51eb34e37c357d751279f080e7da4"
Oct 30 20:02:01.860320 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.860300    4823 scope.go:122] "RemoveContainer" containerID="42fe5f7dfa46a2f7095fc5570aa73f8e227730f470bb967e9ad37f9742a10c3c"
Oct 30 20:02:01.870389 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.870363    4823 scope.go:122] "RemoveContainer" containerID="d2eb316fb78dd12d3a78b03a3a51e8633c8f3cb5ad9f1417fd59ca126eb56049"
Oct 30 20:02:01.880552 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.880528    4823 scope.go:122] "RemoveContainer" containerID="057cb99d26bd8d39bf792ed6c7300c61432f51b9f682858c16cf2152d9460684"
Oct 30 20:02:01.887029 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.886995    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxpzx\" (UniqueName: \"kubernetes.io/projected/f6660e61-7ad3-45cd-873e-d4c0a1741317-kube-api-access-dxpzx\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:01.887158 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.887032    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:01.887158 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.887051    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:01.887158 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.887065    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6660e61-7ad3-45cd-873e-d4c0a1741317-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:01.887158 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.887079    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6660e61-7ad3-45cd-873e-d4c0a1741317-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:01.891609 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.891585    4823 scope.go:122] "RemoveContainer" containerID="86139569599fb137c7e8e6a9225968db289ddc573f9de8674a5e34c98db59a4b"
Oct 30 20:02:01.901741 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:01.901724    4823 scope.go:122] "RemoveContainer" containerID="5b60cfc81b0191dffc2ea9b058439c3d8b91b250b42c2b8bd540159e98d96fb7"
Oct 30 20:02:02.016318 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:02:02.016286    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:02:02.844506 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:02.844052    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"
Oct 30 20:02:03.170885 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:03.169911    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"]
Oct 30 20:02:03.207947 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:03.206757    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-98a0d8a9-openshift-default-db789f9d9-k9jmx"]
Oct 30 20:02:04.396930 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:04.396890    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6660e61-7ad3-45cd-873e-d4c0a1741317" path="/var/lib/kubelet/pods/f6660e61-7ad3-45cd-873e-d4c0a1741317/volumes"
Oct 30 20:02:04.756971 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:04.756845    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"]
Oct 30 20:02:04.763341 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:04.763306    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c4c072-knps9"]
Oct 30 20:02:06.281371 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:06.281339    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5f003b6-482e-4099-94af-1853828bdd2f" path="/var/lib/kubelet/pods/c5f003b6-482e-4099-94af-1853828bdd2f/volumes"
Oct 30 20:02:07.433445 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:07.433415    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"]
Oct 30 20:02:07.438223 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:07.438193    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1b04-zg8bf"]
Oct 30 20:02:07.553632 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:07.553598    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:02:08.281035 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:08.281003    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e82b58ca-46a9-49a4-84a8-38d75eb1048b" path="/var/lib/kubelet/pods/e82b58ca-46a9-49a4-84a8-38d75eb1048b/volumes"
Oct 30 20:02:10.087912 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:10.083817    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:10.139914 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:10.099030    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 30 20:02:11.593336 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.593302    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"]
Oct 30 20:02:11.594235 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.594217    4823 cpu_manager.go:414] "containerMap: removing container" podUID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerName="istio-proxy"
Oct 30 20:02:11.594294 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.594236    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerName="istio-proxy"
Oct 30 20:02:11.594294 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.594252    4823 cpu_manager.go:414] "containerMap: removing container" podUID="c7558ea4-5322-443f-9847-857688238559" containerName="revoke-keys"
Oct 30 20:02:11.594294 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.594258    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7558ea4-5322-443f-9847-857688238559" containerName="revoke-keys"
Oct 30 20:02:11.594438 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.594428    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7558ea4-5322-443f-9847-857688238559" containerName="revoke-keys"
Oct 30 20:02:11.594474 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.594446    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerName="istio-proxy"
Oct 30 20:02:11.600901 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.600877    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"
Oct 30 20:02:11.607661 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.607638    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:02:11.608086 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.608064    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:02:11.613505 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.613477    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"]
Oct 30 20:02:11.675704 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.675656    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl\" (UID: \"6654a377-ad63-4c0a-bf8a-d358e3f56ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"
Oct 30 20:02:11.675994 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.675762    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss9dk\" (UniqueName: \"kubernetes.io/projected/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-kube-api-access-ss9dk\") pod \"maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl\" (UID: \"6654a377-ad63-4c0a-bf8a-d358e3f56ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"
Oct 30 20:02:11.675994 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.675927    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl\" (UID: \"6654a377-ad63-4c0a-bf8a-d358e3f56ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"
Oct 30 20:02:11.778225 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.778164    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl\" (UID: \"6654a377-ad63-4c0a-bf8a-d358e3f56ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"
Oct 30 20:02:11.778543 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.778513    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl\" (UID: \"6654a377-ad63-4c0a-bf8a-d358e3f56ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"
Oct 30 20:02:11.778633 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.778599    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss9dk\" (UniqueName: \"kubernetes.io/projected/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-kube-api-access-ss9dk\") pod \"maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl\" (UID: \"6654a377-ad63-4c0a-bf8a-d358e3f56ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"
Oct 30 20:02:11.779313 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.779286    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl\" (UID: \"6654a377-ad63-4c0a-bf8a-d358e3f56ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"
Oct 30 20:02:11.781127 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.781104    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl\" (UID: \"6654a377-ad63-4c0a-bf8a-d358e3f56ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"
Oct 30 20:02:11.792373 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.792348    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss9dk\" (UniqueName: \"kubernetes.io/projected/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-kube-api-access-ss9dk\") pod \"maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl\" (UID: \"6654a377-ad63-4c0a-bf8a-d358e3f56ad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"
Oct 30 20:02:11.917798 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:11.917763    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"
Oct 30 20:02:12.245554 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:12.245522    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"]
Oct 30 20:02:13.238833 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:13.238792    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl" event={"ID":"6654a377-ad63-4c0a-bf8a-d358e3f56ad4","Type":"ContainerStarted","Data":"30429f5299a135d28a54331885cbd1447337b09a218af9c4ede8a6ce09c7d2c8"}
Oct 30 20:02:13.238833 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:13.238915    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl" event={"ID":"6654a377-ad63-4c0a-bf8a-d358e3f56ad4","Type":"ContainerStarted","Data":"4c16a54b7fa2e43d56168d9027614a3b7899fec8a4d8bf6dfde03bbe43c55d94"}
Oct 30 20:02:13.286197 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:13.286135    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl" podStartSLOduration=2.286120883 podStartE2EDuration="2.286120883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:13.28120116 +0000 UTC m=+1933.173815481" watchObservedRunningTime="2026-10-30 20:02:13.286120883 +0000 UTC m=+1933.178735207"
Oct 30 20:02:13.426884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:13.415204    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"]
Oct 30 20:02:13.426884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:13.415525    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5" podUID="c51ac612-1da2-4152-978a-78726d5ea888" containerName="istio-proxy" containerID="cri-o://36eb5db80d327409de404db63914864ff8ac06cffad65c9cc91c31cfdbaeabc0" gracePeriod=30
Oct 30 20:02:14.082619 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:14.082584    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"]
Oct 30 20:02:14.087536 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:14.087509    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0706d071-l7m89"]
Oct 30 20:02:14.283224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:14.283159    4823 generic.go:356] "Generic (PLEG): container finished" podID="6654a377-ad63-4c0a-bf8a-d358e3f56ad4" containerID="30429f5299a135d28a54331885cbd1447337b09a218af9c4ede8a6ce09c7d2c8" exitCode=0
Oct 30 20:02:14.294477 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:14.294447    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bed2ee64-b400-40ac-bc9f-24ecb430b169" path="/var/lib/kubelet/pods/bed2ee64-b400-40ac-bc9f-24ecb430b169/volumes"
Oct 30 20:02:14.294821 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:14.294802    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl" event={"ID":"6654a377-ad63-4c0a-bf8a-d358e3f56ad4","Type":"ContainerDied","Data":"30429f5299a135d28a54331885cbd1447337b09a218af9c4ede8a6ce09c7d2c8"}
Oct 30 20:02:15.909500 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:15.909470    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"
Oct 30 20:02:16.002379 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:16.002339    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-kube-api-access-ss9dk\" (UniqueName: \"kubernetes.io/projected/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-kube-api-access-ss9dk\") pod \"6654a377-ad63-4c0a-bf8a-d358e3f56ad4\" (UID: \"6654a377-ad63-4c0a-bf8a-d358e3f56ad4\") "
Oct 30 20:02:16.002612 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:16.002413    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-maas-api-xks-ca\") pod \"6654a377-ad63-4c0a-bf8a-d358e3f56ad4\" (UID: \"6654a377-ad63-4c0a-bf8a-d358e3f56ad4\") "
Oct 30 20:02:16.002696 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:16.002611    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-maas-api-openshift-ca\") pod \"6654a377-ad63-4c0a-bf8a-d358e3f56ad4\" (UID: \"6654a377-ad63-4c0a-bf8a-d358e3f56ad4\") "
Oct 30 20:02:16.003502 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:16.003467    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-maas-api-openshift-ca" pod "6654a377-ad63-4c0a-bf8a-d358e3f56ad4" (UID: "6654a377-ad63-4c0a-bf8a-d358e3f56ad4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:16.004614 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:16.004588    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:16.005268 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:16.005232    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-maas-api-xks-ca" pod "6654a377-ad63-4c0a-bf8a-d358e3f56ad4" (UID: "6654a377-ad63-4c0a-bf8a-d358e3f56ad4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:16.005575 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:16.005541    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-kube-api-access-ss9dk" pod "6654a377-ad63-4c0a-bf8a-d358e3f56ad4" (UID: "6654a377-ad63-4c0a-bf8a-d358e3f56ad4"). InnerVolumeSpecName "kube-api-access-ss9dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:16.106266 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:16.106236    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss9dk\" (UniqueName: \"kubernetes.io/projected/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-kube-api-access-ss9dk\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:16.106413 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:16.106274    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6654a377-ad63-4c0a-bf8a-d358e3f56ad4-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:16.497022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:16.488093    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl" event={"ID":"6654a377-ad63-4c0a-bf8a-d358e3f56ad4","Type":"ContainerDied","Data":"4c16a54b7fa2e43d56168d9027614a3b7899fec8a4d8bf6dfde03bbe43c55d94"}
Oct 30 20:02:16.497022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:16.488231    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c16a54b7fa2e43d56168d9027614a3b7899fec8a4d8bf6dfde03bbe43c55d94"
Oct 30 20:02:16.497022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:16.488314    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"
Oct 30 20:02:16.790744 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:16.790095    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"]
Oct 30 20:02:16.790744 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:16.790373    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerName="istio-proxy" containerID="cri-o://878150ec1187b9e42f0aec4c8afff83f8da2668f29faa5b0a97778f9a41711c7" gracePeriod=30
Oct 30 20:02:17.335191 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:17.335154    4823 patch_prober.go:29] interesting pod/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:02:17.335568 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:17.335215    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5" podUID="c51ac612-1da2-4152-978a-78726d5ea888" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:02:18.711941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:18.656358    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"]
Oct 30 20:02:18.711941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:18.656700    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp" podUID="80073467-436a-4974-b535-170e24af7092" containerName="istio-proxy" containerID="cri-o://a9d5f284d96f4002c4a72df39d17c2be2af5a9a1350b2be90103d079afbfb3bd" gracePeriod=30
Oct 30 20:02:19.274683 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.274654    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 20:02:19.448903 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.448787    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-credential-socket\") pod \"c51ac612-1da2-4152-978a-78726d5ea888\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") "
Oct 30 20:02:19.449091 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.448924    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51ac612-1da2-4152-978a-78726d5ea888-istio-token\" (UniqueName: \"kubernetes.io/projected/c51ac612-1da2-4152-978a-78726d5ea888-istio-token\") pod \"c51ac612-1da2-4152-978a-78726d5ea888\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") "
Oct 30 20:02:19.449091 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.448998    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-workload-certs\") pod \"c51ac612-1da2-4152-978a-78726d5ea888\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") "
Oct 30 20:02:19.449091 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.449030    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-istio-envoy\") pod \"c51ac612-1da2-4152-978a-78726d5ea888\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") "
Oct 30 20:02:19.449262 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.449169    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-credential-socket" pod "c51ac612-1da2-4152-978a-78726d5ea888" (UID: "c51ac612-1da2-4152-978a-78726d5ea888"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:19.449262 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.449224    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51ac612-1da2-4152-978a-78726d5ea888-kube-api-access-r7r85\" (UniqueName: \"kubernetes.io/projected/c51ac612-1da2-4152-978a-78726d5ea888-kube-api-access-r7r85\") pod \"c51ac612-1da2-4152-978a-78726d5ea888\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") "
Oct 30 20:02:19.449377 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.449319    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51ac612-1da2-4152-978a-78726d5ea888-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c51ac612-1da2-4152-978a-78726d5ea888-istiod-ca-cert\") pod \"c51ac612-1da2-4152-978a-78726d5ea888\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") "
Oct 30 20:02:19.449436 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.449391    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-istio-data\") pod \"c51ac612-1da2-4152-978a-78726d5ea888\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") "
Oct 30 20:02:19.449504 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.449446    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c51ac612-1da2-4152-978a-78726d5ea888-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c51ac612-1da2-4152-978a-78726d5ea888-istio-podinfo\") pod \"c51ac612-1da2-4152-978a-78726d5ea888\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") "
Oct 30 20:02:19.449564 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.449554    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-workload-socket\") pod \"c51ac612-1da2-4152-978a-78726d5ea888\" (UID: \"c51ac612-1da2-4152-978a-78726d5ea888\") "
Oct 30 20:02:19.449856 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.449699    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-workload-certs" pod "c51ac612-1da2-4152-978a-78726d5ea888" (UID: "c51ac612-1da2-4152-978a-78726d5ea888"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:19.450751 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.450722    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:19.450959 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.450756    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:19.451349 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.451318    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-istio-data" pod "c51ac612-1da2-4152-978a-78726d5ea888" (UID: "c51ac612-1da2-4152-978a-78726d5ea888"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:19.452251 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.452223    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-workload-socket" pod "c51ac612-1da2-4152-978a-78726d5ea888" (UID: "c51ac612-1da2-4152-978a-78726d5ea888"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:19.452441 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.452420    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51ac612-1da2-4152-978a-78726d5ea888-istiod-ca-cert" pod "c51ac612-1da2-4152-978a-78726d5ea888" (UID: "c51ac612-1da2-4152-978a-78726d5ea888"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:19.453427 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.453383    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-istio-envoy" pod "c51ac612-1da2-4152-978a-78726d5ea888" (UID: "c51ac612-1da2-4152-978a-78726d5ea888"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:19.453427 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.453404    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c51ac612-1da2-4152-978a-78726d5ea888-istio-podinfo" pod "c51ac612-1da2-4152-978a-78726d5ea888" (UID: "c51ac612-1da2-4152-978a-78726d5ea888"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:19.453605 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.453583    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51ac612-1da2-4152-978a-78726d5ea888-kube-api-access-r7r85" pod "c51ac612-1da2-4152-978a-78726d5ea888" (UID: "c51ac612-1da2-4152-978a-78726d5ea888"). InnerVolumeSpecName "kube-api-access-r7r85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:19.453709 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.453691    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51ac612-1da2-4152-978a-78726d5ea888-istio-token" pod "c51ac612-1da2-4152-978a-78726d5ea888" (UID: "c51ac612-1da2-4152-978a-78726d5ea888"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:19.552605 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.552562    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7r85\" (UniqueName: \"kubernetes.io/projected/c51ac612-1da2-4152-978a-78726d5ea888-kube-api-access-r7r85\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:19.552605 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.552599    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c51ac612-1da2-4152-978a-78726d5ea888-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:19.552605 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.552612    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:19.552953 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.552620    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c51ac612-1da2-4152-978a-78726d5ea888-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:19.552953 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.552629    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:19.552953 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.552638    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c51ac612-1da2-4152-978a-78726d5ea888-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:19.552953 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.552646    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c51ac612-1da2-4152-978a-78726d5ea888-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:19.941508 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.941464    4823 generic.go:356] "Generic (PLEG): container finished" podID="c51ac612-1da2-4152-978a-78726d5ea888" containerID="36eb5db80d327409de404db63914864ff8ac06cffad65c9cc91c31cfdbaeabc0" exitCode=0
Oct 30 20:02:19.942008 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.941527    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"
Oct 30 20:02:19.942008 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.941602    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5" event={"ID":"c51ac612-1da2-4152-978a-78726d5ea888","Type":"ContainerDied","Data":"36eb5db80d327409de404db63914864ff8ac06cffad65c9cc91c31cfdbaeabc0"}
Oct 30 20:02:19.942008 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.941658    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5" event={"ID":"c51ac612-1da2-4152-978a-78726d5ea888","Type":"ContainerDied","Data":"82fa69f4d7fa880bde3651042944cca0315ce3479108855f9768e8a8ffffb922"}
Oct 30 20:02:19.942008 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.941680    4823 scope.go:122] "RemoveContainer" containerID="36eb5db80d327409de404db63914864ff8ac06cffad65c9cc91c31cfdbaeabc0"
Oct 30 20:02:19.962586 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.961306    4823 scope.go:122] "RemoveContainer" containerID="f08e2a7bdcb16b2064e9884c5ef38f7c4cc455ef3b3e02e890d2675e7664ed84"
Oct 30 20:02:19.972599 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.972571    4823 scope.go:122] "RemoveContainer" containerID="36eb5db80d327409de404db63914864ff8ac06cffad65c9cc91c31cfdbaeabc0"
Oct 30 20:02:19.972913 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:02:19.972873    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36eb5db80d327409de404db63914864ff8ac06cffad65c9cc91c31cfdbaeabc0\": container with ID starting with 36eb5db80d327409de404db63914864ff8ac06cffad65c9cc91c31cfdbaeabc0 not found: ID does not exist" containerID="36eb5db80d327409de404db63914864ff8ac06cffad65c9cc91c31cfdbaeabc0"
Oct 30 20:02:19.972913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.972906    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36eb5db80d327409de404db63914864ff8ac06cffad65c9cc91c31cfdbaeabc0"} err="failed to get container status \"36eb5db80d327409de404db63914864ff8ac06cffad65c9cc91c31cfdbaeabc0\": rpc error: code = NotFound desc = could not find container \"36eb5db80d327409de404db63914864ff8ac06cffad65c9cc91c31cfdbaeabc0\": container with ID starting with 36eb5db80d327409de404db63914864ff8ac06cffad65c9cc91c31cfdbaeabc0 not found: ID does not exist"
Oct 30 20:02:19.973090 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.972923    4823 scope.go:122] "RemoveContainer" containerID="f08e2a7bdcb16b2064e9884c5ef38f7c4cc455ef3b3e02e890d2675e7664ed84"
Oct 30 20:02:19.973209 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:02:19.973191    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08e2a7bdcb16b2064e9884c5ef38f7c4cc455ef3b3e02e890d2675e7664ed84\": container with ID starting with f08e2a7bdcb16b2064e9884c5ef38f7c4cc455ef3b3e02e890d2675e7664ed84 not found: ID does not exist" containerID="f08e2a7bdcb16b2064e9884c5ef38f7c4cc455ef3b3e02e890d2675e7664ed84"
Oct 30 20:02:19.973268 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.973211    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08e2a7bdcb16b2064e9884c5ef38f7c4cc455ef3b3e02e890d2675e7664ed84"} err="failed to get container status \"f08e2a7bdcb16b2064e9884c5ef38f7c4cc455ef3b3e02e890d2675e7664ed84\": rpc error: code = NotFound desc = could not find container \"f08e2a7bdcb16b2064e9884c5ef38f7c4cc455ef3b3e02e890d2675e7664ed84\": container with ID starting with f08e2a7bdcb16b2064e9884c5ef38f7c4cc455ef3b3e02e890d2675e7664ed84 not found: ID does not exist"
Oct 30 20:02:20.000882 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:19.998516    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"]
Oct 30 20:02:20.009883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.005585    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c4c072-openshift-default-67644f876f-nbtj5"]
Oct 30 20:02:20.013736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.013712    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"]
Oct 30 20:02:20.014550 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.014531    4823 cpu_manager.go:414] "containerMap: removing container" podUID="6654a377-ad63-4c0a-bf8a-d358e3f56ad4" containerName="revoke-keys"
Oct 30 20:02:20.014550 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.014550    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6654a377-ad63-4c0a-bf8a-d358e3f56ad4" containerName="revoke-keys"
Oct 30 20:02:20.014715 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.014578    4823 cpu_manager.go:414] "containerMap: removing container" podUID="c51ac612-1da2-4152-978a-78726d5ea888" containerName="istio-proxy"
Oct 30 20:02:20.014715 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.014583    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51ac612-1da2-4152-978a-78726d5ea888" containerName="istio-proxy"
Oct 30 20:02:20.014715 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.014591    4823 cpu_manager.go:414] "containerMap: removing container" podUID="c51ac612-1da2-4152-978a-78726d5ea888" containerName="istio-proxy"
Oct 30 20:02:20.014715 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.014596    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51ac612-1da2-4152-978a-78726d5ea888" containerName="istio-proxy"
Oct 30 20:02:20.014715 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.014603    4823 cpu_manager.go:414] "containerMap: removing container" podUID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerName="istio-proxy"
Oct 30 20:02:20.014715 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.014608    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerName="istio-proxy"
Oct 30 20:02:20.015077 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.014730    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51ac612-1da2-4152-978a-78726d5ea888" containerName="istio-proxy"
Oct 30 20:02:20.015077 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.014748    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6660e61-7ad3-45cd-873e-d4c0a1741317" containerName="istio-proxy"
Oct 30 20:02:20.015077 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.014757    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="6654a377-ad63-4c0a-bf8a-d358e3f56ad4" containerName="revoke-keys"
Oct 30 20:02:20.020674 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.020653    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.025648 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.025612    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3a4f02-openshift-default-dockercfg-89zh4\""
Oct 30 20:02:20.047510 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.047484    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"]
Oct 30 20:02:20.166427 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.166394    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-token\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.166602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.166455    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istiod-ca-cert\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.166602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.166565    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-podinfo\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.166713 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.166600    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-envoy\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.166713 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.166623    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-workload-certs\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.166792 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.166715    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-credential-socket\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.166966 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.166947    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-data\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.167080 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.167064    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7f2h\" (UniqueName: \"kubernetes.io/projected/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-kube-api-access-j7f2h\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.167142 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.167106    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-workload-socket\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.285938 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.285432    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-token\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.285938 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.285512    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istiod-ca-cert\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.285938 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.285588    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-podinfo\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.285938 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.285620    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-envoy\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.285938 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.285653    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-workload-certs\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.285938 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.285683    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-credential-socket\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.285938 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.285903    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-data\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.286464 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.286007    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7f2h\" (UniqueName: \"kubernetes.io/projected/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-kube-api-access-j7f2h\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.286464 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.286049    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-workload-socket\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.297891 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.286604    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-workload-socket\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.297891 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.287619    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istiod-ca-cert\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.331884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.297189    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-podinfo\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.331884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.303842    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-data\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.331884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.310948    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-credential-socket\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.331884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.311226    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-workload-certs\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.336039 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.335964    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-envoy\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.366993 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.340687    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-token\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.366993 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.365089    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7f2h\" (UniqueName: \"kubernetes.io/projected/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-kube-api-access-j7f2h\") pod \"e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") " pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.527886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.522690    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51ac612-1da2-4152-978a-78726d5ea888" path="/var/lib/kubelet/pods/c51ac612-1da2-4152-978a-78726d5ea888/volumes"
Oct 30 20:02:20.644384 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.644352    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:20.970785 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.970755    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"]
Oct 30 20:02:20.974767 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.974712    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:02:20.974857 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.974824    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:02:20.974923 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:20.974896    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:02:21.971549 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:21.971507    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n" event={"ID":"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3","Type":"ContainerStarted","Data":"895393ad8bd262e70e9d798e3795bb459ae01e232eee17b1cc38488951747dcb"}
Oct 30 20:02:21.972062 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:21.971578    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n" event={"ID":"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3","Type":"ContainerStarted","Data":"79509709aa674380892d2d467c7ac2d1d2a893a936a003a485b2bce15df3b85a"}
Oct 30 20:02:21.974572 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:21.974542    4823 generic.go:356] "Generic (PLEG): container finished" podID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerID="878150ec1187b9e42f0aec4c8afff83f8da2668f29faa5b0a97778f9a41711c7" exitCode=0
Oct 30 20:02:21.974714 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:21.974590    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" event={"ID":"aa02d0d0-81a7-4a03-9114-6561a01c3ffb","Type":"ContainerDied","Data":"878150ec1187b9e42f0aec4c8afff83f8da2668f29faa5b0a97778f9a41711c7"}
Oct 30 20:02:21.974714 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:21.974642    4823 scope.go:122] "RemoveContainer" containerID="05a5e55cf6ae8b74925442f2851a2269a207f30441962c7467a853302a5f188a"
Oct 30 20:02:22.009130 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.009071    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n" podStartSLOduration=3.009052634 podStartE2EDuration="3.009052634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:22.002384682 +0000 UTC m=+1941.894999011" watchObservedRunningTime="2026-10-30 20:02:22.009052634 +0000 UTC m=+1941.901666959"
Oct 30 20:02:22.082116 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.082049    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 20:02:22.145044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.115345    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-podinfo\") pod \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") "
Oct 30 20:02:22.145044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.115423    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-envoy\") pod \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") "
Oct 30 20:02:22.145044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.115466    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-workload-socket\") pod \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") "
Oct 30 20:02:22.145044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.115493    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istiod-ca-cert\") pod \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") "
Oct 30 20:02:22.145044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.115521    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-workload-certs\") pod \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") "
Oct 30 20:02:22.145044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.115688    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-credential-socket\") pod \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") "
Oct 30 20:02:22.145044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.115820    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-data\") pod \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") "
Oct 30 20:02:22.145044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.115855    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-token\" (UniqueName: \"kubernetes.io/projected/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-token\") pod \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") "
Oct 30 20:02:22.145044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.116075    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-kube-api-access-lrqrw\" (UniqueName: \"kubernetes.io/projected/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-kube-api-access-lrqrw\") pod \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\" (UID: \"aa02d0d0-81a7-4a03-9114-6561a01c3ffb\") "
Oct 30 20:02:22.145044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.117558    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istiod-ca-cert" pod "aa02d0d0-81a7-4a03-9114-6561a01c3ffb" (UID: "aa02d0d0-81a7-4a03-9114-6561a01c3ffb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:22.145044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.119082    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-workload-certs" pod "aa02d0d0-81a7-4a03-9114-6561a01c3ffb" (UID: "aa02d0d0-81a7-4a03-9114-6561a01c3ffb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:22.145044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.119152    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-data" pod "aa02d0d0-81a7-4a03-9114-6561a01c3ffb" (UID: "aa02d0d0-81a7-4a03-9114-6561a01c3ffb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:22.145044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.119459    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-workload-socket" pod "aa02d0d0-81a7-4a03-9114-6561a01c3ffb" (UID: "aa02d0d0-81a7-4a03-9114-6561a01c3ffb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:22.145044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.119506    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-credential-socket" pod "aa02d0d0-81a7-4a03-9114-6561a01c3ffb" (UID: "aa02d0d0-81a7-4a03-9114-6561a01c3ffb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:22.159886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.156196    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-token" pod "aa02d0d0-81a7-4a03-9114-6561a01c3ffb" (UID: "aa02d0d0-81a7-4a03-9114-6561a01c3ffb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:22.208887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.171055    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-kube-api-access-lrqrw" pod "aa02d0d0-81a7-4a03-9114-6561a01c3ffb" (UID: "aa02d0d0-81a7-4a03-9114-6561a01c3ffb"). InnerVolumeSpecName "kube-api-access-lrqrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:22.208887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.172119    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-podinfo" pod "aa02d0d0-81a7-4a03-9114-6561a01c3ffb" (UID: "aa02d0d0-81a7-4a03-9114-6561a01c3ffb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:22.208887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.177025    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-envoy" pod "aa02d0d0-81a7-4a03-9114-6561a01c3ffb" (UID: "aa02d0d0-81a7-4a03-9114-6561a01c3ffb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:22.571884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.517652    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:22.571884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.540206    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrqrw\" (UniqueName: \"kubernetes.io/projected/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-kube-api-access-lrqrw\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:22.571884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.540243    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:22.571884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.540259    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:22.571884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.540274    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:22.571884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.540290    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:22.571884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.540305    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:22.571884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.540320    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:22.571884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.540335    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa02d0d0-81a7-4a03-9114-6561a01c3ffb-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:22.646052 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.644442    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:22.680430 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.680377    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:22.775456 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:02:22.775420    4823 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa02d0d0_81a7_4a03_9114_6561a01c3ffb.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa02d0d0_81a7_4a03_9114_6561a01c3ffb.slice/crio-1dda84c3d19cabc59314d3af6e4768a73ef32b3716b1d2d271bc2074dea2163a\": RecentStats: unable to find data in memory cache]"
Oct 30 20:02:22.987934 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.987831    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"
Oct 30 20:02:22.987934 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.987912    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96" event={"ID":"aa02d0d0-81a7-4a03-9114-6561a01c3ffb","Type":"ContainerDied","Data":"1dda84c3d19cabc59314d3af6e4768a73ef32b3716b1d2d271bc2074dea2163a"}
Oct 30 20:02:22.988493 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.987991    4823 scope.go:122] "RemoveContainer" containerID="878150ec1187b9e42f0aec4c8afff83f8da2668f29faa5b0a97778f9a41711c7"
Oct 30 20:02:22.988655 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.988634    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:22.990817 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:22.990791    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:02:23.046458 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:23.046205    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"]
Oct 30 20:02:23.051543 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:23.051506    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ae43d70e-openshift-default-5df45f6c97-5bg96"]
Oct 30 20:02:24.001457 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.001429    4823 generic.go:356] "Generic (PLEG): container finished" podID="35ad66e7-5505-4c0b-96f8-833adf2bd769" containerID="dccc357281ebafd063b095e06fb81e77a6fc66b00a9cd7a558b5ab49f7f09eb9" exitCode=0
Oct 30 20:02:24.001910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.001506    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9" event={"ID":"35ad66e7-5505-4c0b-96f8-833adf2bd769","Type":"ContainerDied","Data":"dccc357281ebafd063b095e06fb81e77a6fc66b00a9cd7a558b5ab49f7f09eb9"}
Oct 30 20:02:24.002042 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.002022    4823 scope.go:122] "RemoveContainer" containerID="dccc357281ebafd063b095e06fb81e77a6fc66b00a9cd7a558b5ab49f7f09eb9"
Oct 30 20:02:24.003780 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.003604    4823 generic.go:356] "Generic (PLEG): container finished" podID="80073467-436a-4974-b535-170e24af7092" containerID="a9d5f284d96f4002c4a72df39d17c2be2af5a9a1350b2be90103d079afbfb3bd" exitCode=0
Oct 30 20:02:24.004645 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.004551    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp" event={"ID":"80073467-436a-4974-b535-170e24af7092","Type":"ContainerDied","Data":"a9d5f284d96f4002c4a72df39d17c2be2af5a9a1350b2be90103d079afbfb3bd"}
Oct 30 20:02:24.006325 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.006292    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:02:24.006417 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.006378    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:02:24.006483 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.006434    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:02:24.328730 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.324352    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" path="/var/lib/kubelet/pods/aa02d0d0-81a7-4a03-9114-6561a01c3ffb/volumes"
Oct 30 20:02:24.709099 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.709074    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:02:24.805309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.805099    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80073467-436a-4974-b535-170e24af7092-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80073467-436a-4974-b535-170e24af7092-istiod-ca-cert\") pod \"80073467-436a-4974-b535-170e24af7092\" (UID: \"80073467-436a-4974-b535-170e24af7092\") "
Oct 30 20:02:24.805309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.805176    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-istio-data\") pod \"80073467-436a-4974-b535-170e24af7092\" (UID: \"80073467-436a-4974-b535-170e24af7092\") "
Oct 30 20:02:24.805554 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.805346    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-istio-envoy\") pod \"80073467-436a-4974-b535-170e24af7092\" (UID: \"80073467-436a-4974-b535-170e24af7092\") "
Oct 30 20:02:24.805554 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.805444    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-workload-certs\") pod \"80073467-436a-4974-b535-170e24af7092\" (UID: \"80073467-436a-4974-b535-170e24af7092\") "
Oct 30 20:02:24.805554 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.805524    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-workload-socket\") pod \"80073467-436a-4974-b535-170e24af7092\" (UID: \"80073467-436a-4974-b535-170e24af7092\") "
Oct 30 20:02:24.805710 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.805558    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80073467-436a-4974-b535-170e24af7092-istiod-ca-cert" pod "80073467-436a-4974-b535-170e24af7092" (UID: "80073467-436a-4974-b535-170e24af7092"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:24.805710 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.805565    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-credential-socket\") pod \"80073467-436a-4974-b535-170e24af7092\" (UID: \"80073467-436a-4974-b535-170e24af7092\") "
Oct 30 20:02:24.805710 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.805614    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80073467-436a-4974-b535-170e24af7092-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80073467-436a-4974-b535-170e24af7092-istio-podinfo\") pod \"80073467-436a-4974-b535-170e24af7092\" (UID: \"80073467-436a-4974-b535-170e24af7092\") "
Oct 30 20:02:24.805710 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.805695    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80073467-436a-4974-b535-170e24af7092-kube-api-access-wrkj5\" (UniqueName: \"kubernetes.io/projected/80073467-436a-4974-b535-170e24af7092-kube-api-access-wrkj5\") pod \"80073467-436a-4974-b535-170e24af7092\" (UID: \"80073467-436a-4974-b535-170e24af7092\") "
Oct 30 20:02:24.805948 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.805736    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80073467-436a-4974-b535-170e24af7092-istio-token\" (UniqueName: \"kubernetes.io/projected/80073467-436a-4974-b535-170e24af7092-istio-token\") pod \"80073467-436a-4974-b535-170e24af7092\" (UID: \"80073467-436a-4974-b535-170e24af7092\") "
Oct 30 20:02:24.806116 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.806089    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-istio-data" pod "80073467-436a-4974-b535-170e24af7092" (UID: "80073467-436a-4974-b535-170e24af7092"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.806369 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.806343    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-credential-socket" pod "80073467-436a-4974-b535-170e24af7092" (UID: "80073467-436a-4974-b535-170e24af7092"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.806848 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.806820    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.807016 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.806882    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80073467-436a-4974-b535-170e24af7092-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.807016 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.806911    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.807232 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.807203    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-workload-certs" pod "80073467-436a-4974-b535-170e24af7092" (UID: "80073467-436a-4974-b535-170e24af7092"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.807398 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.807276    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-workload-socket" pod "80073467-436a-4974-b535-170e24af7092" (UID: "80073467-436a-4974-b535-170e24af7092"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.809058 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.809026    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80073467-436a-4974-b535-170e24af7092-istio-podinfo" pod "80073467-436a-4974-b535-170e24af7092" (UID: "80073467-436a-4974-b535-170e24af7092"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:24.809447 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.809422    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-istio-envoy" pod "80073467-436a-4974-b535-170e24af7092" (UID: "80073467-436a-4974-b535-170e24af7092"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:24.810437 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.809738    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80073467-436a-4974-b535-170e24af7092-kube-api-access-wrkj5" pod "80073467-436a-4974-b535-170e24af7092" (UID: "80073467-436a-4974-b535-170e24af7092"). InnerVolumeSpecName "kube-api-access-wrkj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:24.811327 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.811298    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80073467-436a-4974-b535-170e24af7092-istio-token" pod "80073467-436a-4974-b535-170e24af7092" (UID: "80073467-436a-4974-b535-170e24af7092"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:24.908485 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.908456    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.908485 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.908482    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.908698 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.908495    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80073467-436a-4974-b535-170e24af7092-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.908698 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.908505    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrkj5\" (UniqueName: \"kubernetes.io/projected/80073467-436a-4974-b535-170e24af7092-kube-api-access-wrkj5\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.908698 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.908515    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80073467-436a-4974-b535-170e24af7092-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:24.908698 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:24.908524    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80073467-436a-4974-b535-170e24af7092-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:25.015961 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.015878    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9" event={"ID":"35ad66e7-5505-4c0b-96f8-833adf2bd769","Type":"ContainerStarted","Data":"522c0e6ae0d7d02cafbaaba4eccb9e8c43ff52315c37ae185e22eb4833b4fd9d"}
Oct 30 20:02:25.017585 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.017563    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"
Oct 30 20:02:25.017585 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.017579    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp" event={"ID":"80073467-436a-4974-b535-170e24af7092","Type":"ContainerDied","Data":"8be0bdf468adad2c210276f17ac28d3aa1f927279802a3d32b4a0f6315ae112b"}
Oct 30 20:02:25.018435 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.017629    4823 scope.go:122] "RemoveContainer" containerID="a9d5f284d96f4002c4a72df39d17c2be2af5a9a1350b2be90103d079afbfb3bd"
Oct 30 20:02:25.071511 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.071474    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"]
Oct 30 20:02:25.077126 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.077102    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-zp5hp"]
Oct 30 20:02:25.340779 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.340695    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:02:25.342430 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.342396    4823 patch_prober.go:29] interesting pod/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:02:25.342577 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.342457    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9" podUID="35ad66e7-5505-4c0b-96f8-833adf2bd769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 20:02:25.797621 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.797576    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"]
Oct 30 20:02:25.800421 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.800384    4823 cpu_manager.go:414] "containerMap: removing container" podUID="80073467-436a-4974-b535-170e24af7092" containerName="istio-proxy"
Oct 30 20:02:25.800421 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.800421    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80073467-436a-4974-b535-170e24af7092" containerName="istio-proxy"
Oct 30 20:02:25.800587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.800493    4823 cpu_manager.go:414] "containerMap: removing container" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerName="istio-proxy"
Oct 30 20:02:25.800587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.800506    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerName="istio-proxy"
Oct 30 20:02:25.800587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.800531    4823 cpu_manager.go:414] "containerMap: removing container" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerName="istio-proxy"
Oct 30 20:02:25.800587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.800539    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerName="istio-proxy"
Oct 30 20:02:25.800989 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.800966    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51ac612-1da2-4152-978a-78726d5ea888" containerName="istio-proxy"
Oct 30 20:02:25.801137 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.801001    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="80073467-436a-4974-b535-170e24af7092" containerName="istio-proxy"
Oct 30 20:02:25.801137 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.801027    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerName="istio-proxy"
Oct 30 20:02:25.801137 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.801045    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa02d0d0-81a7-4a03-9114-6561a01c3ffb" containerName="istio-proxy"
Oct 30 20:02:25.845599 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.829849    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"]
Oct 30 20:02:25.845599 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.830973    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:02:25.884668 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.884634    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1f246b-serving-cert\""
Oct 30 20:02:25.926614 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.926578    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-metrics-certs\") pod \"maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556\" (UID: \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:02:25.926797 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.926741    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556\" (UID: \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:02:25.926841 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.926794    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556\" (UID: \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:02:25.926926 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:25.926902    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv9x9\" (UniqueName: \"kubernetes.io/projected/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-kube-api-access-kv9x9\") pod \"maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556\" (UID: \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:02:26.054796 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:26.036671    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-metrics-certs\") pod \"maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556\" (UID: \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:02:26.124139 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:26.124099    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556\" (UID: \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:02:26.124468 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:26.124446    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556\" (UID: \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:02:26.140887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:26.124589    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv9x9\" (UniqueName: \"kubernetes.io/projected/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-kube-api-access-kv9x9\") pod \"maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556\" (UID: \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:02:26.178887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:26.143232    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556\" (UID: \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:02:26.216881 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:26.211959    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv9x9\" (UniqueName: \"kubernetes.io/projected/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-kube-api-access-kv9x9\") pod \"maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556\" (UID: \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:02:26.292144 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:26.265795    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-metrics-certs\") pod \"maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556\" (UID: \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:02:26.295889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:26.294550    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556\" (UID: \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:02:26.374186 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:26.374142    4823 patch_prober.go:29] interesting pod/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:02:26.374472 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:26.374444    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9" podUID="35ad66e7-5505-4c0b-96f8-833adf2bd769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 20:02:26.416224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:26.416090    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80073467-436a-4974-b535-170e24af7092" path="/var/lib/kubelet/pods/80073467-436a-4974-b535-170e24af7092/volumes"
Oct 30 20:02:26.489924 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:26.489889    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:02:27.087177 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:27.087090    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"]
Oct 30 20:02:27.091750 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:02:27.091720    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c0ca13_508b_45d0_ade5_75dbf9bbde50.slice/crio-18646ef99c9f7c9c81fc8832eddef0c3211943f6fc57f5d2e5454c62f02dd2fa WatchSource:0}: Error finding container 18646ef99c9f7c9c81fc8832eddef0c3211943f6fc57f5d2e5454c62f02dd2fa: Status 404 returned error can't find the container with id 18646ef99c9f7c9c81fc8832eddef0c3211943f6fc57f5d2e5454c62f02dd2fa
Oct 30 20:02:27.342666 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:27.342593    4823 patch_prober.go:29] interesting pod/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:02:27.342816 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:27.342659    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9" podUID="35ad66e7-5505-4c0b-96f8-833adf2bd769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 20:02:27.513887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:27.513707    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556" event={"ID":"d0c0ca13-508b-45d0-ade5-75dbf9bbde50","Type":"ContainerStarted","Data":"18646ef99c9f7c9c81fc8832eddef0c3211943f6fc57f5d2e5454c62f02dd2fa"}
Oct 30 20:02:28.340466 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:28.340431    4823 patch_prober.go:29] interesting pod/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:02:28.340890 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:28.340499    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9" podUID="35ad66e7-5505-4c0b-96f8-833adf2bd769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 20:02:28.526051 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:28.526009    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556" event={"ID":"d0c0ca13-508b-45d0-ade5-75dbf9bbde50","Type":"ContainerStarted","Data":"357e1945030cb62d7c680dafbbd3965ac0bff9269fc23031dbf29421b467fd6b"}
Oct 30 20:02:28.526245 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:28.526147    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:02:28.551702 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:28.551654    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556" podStartSLOduration=2.992466432 podStartE2EDuration="3.551640141s" totalImagesPullingTime="559.173709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:28.54934747 +0000 UTC m=+1948.441961795" watchObservedRunningTime="2026-10-30 20:02:28.551640141 +0000 UTC m=+1948.444254465"
Oct 30 20:02:29.341013 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:29.340976    4823 patch_prober.go:29] interesting pod/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:02:29.341462 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:29.341032    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9" podUID="35ad66e7-5505-4c0b-96f8-833adf2bd769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 20:02:30.341274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:30.341237    4823 patch_prober.go:29] interesting pod/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:02:30.341682 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:30.341289    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9" podUID="35ad66e7-5505-4c0b-96f8-833adf2bd769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 20:02:31.341206 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:31.341168    4823 patch_prober.go:29] interesting pod/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 30 20:02:31.341401 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:31.341223    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9" podUID="35ad66e7-5505-4c0b-96f8-833adf2bd769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 30 20:02:31.365731 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:31.362958    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:02:32.016486 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:02:32.016448    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:02:32.292227 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:32.292145    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"]
Oct 30 20:02:32.292428 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:32.292372    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj" podUID="c2911d92-5384-4bbf-b875-c846a7df7459" containerName="maas-api" containerID="cri-o://872ff9a2d3a8a20bf0ac497fdd92484a6f3f07940f34d614a5247068992318c7" gracePeriod=30
Oct 30 20:02:32.345540 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:32.345514    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:02:32.346573 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:32.346553    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:02:32.795884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:32.791538    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"]
Oct 30 20:02:32.942054 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:32.941242    4823 generic.go:356] "Generic (PLEG): container finished" podID="c2911d92-5384-4bbf-b875-c846a7df7459" containerID="872ff9a2d3a8a20bf0ac497fdd92484a6f3f07940f34d614a5247068992318c7" exitCode=0
Oct 30 20:02:32.942670 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:32.942600    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj" event={"ID":"c2911d92-5384-4bbf-b875-c846a7df7459","Type":"ContainerDied","Data":"872ff9a2d3a8a20bf0ac497fdd92484a6f3f07940f34d614a5247068992318c7"}
Oct 30 20:02:32.984018 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:32.983925    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-35deec-24nt4"]
Oct 30 20:02:33.120379 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.120352    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:02:33.196061 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.196024    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2911d92-5384-4bbf-b875-c846a7df7459-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2911d92-5384-4bbf-b875-c846a7df7459-metrics-certs\") pod \"c2911d92-5384-4bbf-b875-c846a7df7459\" (UID: \"c2911d92-5384-4bbf-b875-c846a7df7459\") "
Oct 30 20:02:33.196338 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.196174    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2911d92-5384-4bbf-b875-c846a7df7459-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2911d92-5384-4bbf-b875-c846a7df7459-maas-api-tls\") pod \"c2911d92-5384-4bbf-b875-c846a7df7459\" (UID: \"c2911d92-5384-4bbf-b875-c846a7df7459\") "
Oct 30 20:02:33.196338 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.196218    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2911d92-5384-4bbf-b875-c846a7df7459-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2911d92-5384-4bbf-b875-c846a7df7459-openshift-service-ca\") pod \"c2911d92-5384-4bbf-b875-c846a7df7459\" (UID: \"c2911d92-5384-4bbf-b875-c846a7df7459\") "
Oct 30 20:02:33.196539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.196329    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2911d92-5384-4bbf-b875-c846a7df7459-kube-api-access-9g27r\" (UniqueName: \"kubernetes.io/projected/c2911d92-5384-4bbf-b875-c846a7df7459-kube-api-access-9g27r\") pod \"c2911d92-5384-4bbf-b875-c846a7df7459\" (UID: \"c2911d92-5384-4bbf-b875-c846a7df7459\") "
Oct 30 20:02:33.197483 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.197450    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2911d92-5384-4bbf-b875-c846a7df7459-openshift-service-ca" pod "c2911d92-5384-4bbf-b875-c846a7df7459" (UID: "c2911d92-5384-4bbf-b875-c846a7df7459"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:33.200258 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.200215    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2911d92-5384-4bbf-b875-c846a7df7459-metrics-certs" pod "c2911d92-5384-4bbf-b875-c846a7df7459" (UID: "c2911d92-5384-4bbf-b875-c846a7df7459"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:33.200378 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.200330    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2911d92-5384-4bbf-b875-c846a7df7459-kube-api-access-9g27r" pod "c2911d92-5384-4bbf-b875-c846a7df7459" (UID: "c2911d92-5384-4bbf-b875-c846a7df7459"). InnerVolumeSpecName "kube-api-access-9g27r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:33.204101 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.204073    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2911d92-5384-4bbf-b875-c846a7df7459-maas-api-tls" pod "c2911d92-5384-4bbf-b875-c846a7df7459" (UID: "c2911d92-5384-4bbf-b875-c846a7df7459"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:33.298957 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.298924    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2911d92-5384-4bbf-b875-c846a7df7459-maas-api-tls\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:33.298957 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.298959    4823 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2911d92-5384-4bbf-b875-c846a7df7459-openshift-service-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:33.299158 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.298976    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g27r\" (UniqueName: \"kubernetes.io/projected/c2911d92-5384-4bbf-b875-c846a7df7459-kube-api-access-9g27r\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:33.299158 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.298994    4823 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2911d92-5384-4bbf-b875-c846a7df7459-metrics-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:33.950898 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.950836    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"
Oct 30 20:02:33.950898 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.950856    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj" event={"ID":"c2911d92-5384-4bbf-b875-c846a7df7459","Type":"ContainerDied","Data":"d153770f1c9e98bcd08be6904357cef15c6f18966f111f5cd099bf20c34a51d6"}
Oct 30 20:02:33.951496 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.950954    4823 scope.go:122] "RemoveContainer" containerID="872ff9a2d3a8a20bf0ac497fdd92484a6f3f07940f34d614a5247068992318c7"
Oct 30 20:02:33.981252 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.981217    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"]
Oct 30 20:02:33.986877 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:33.986842    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-35deec-7d56cdf4-g26hj"]
Oct 30 20:02:34.210066 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.209946    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"]
Oct 30 20:02:34.211522 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.211139    4823 cpu_manager.go:414] "containerMap: removing container" podUID="c2911d92-5384-4bbf-b875-c846a7df7459" containerName="maas-api"
Oct 30 20:02:34.211522 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.211165    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2911d92-5384-4bbf-b875-c846a7df7459" containerName="maas-api"
Oct 30 20:02:34.211522 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.211412    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2911d92-5384-4bbf-b875-c846a7df7459" containerName="maas-api"
Oct 30 20:02:34.216901 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.216853    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.231491 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.231334    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"]
Oct 30 20:02:34.238835 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.238804    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a7bb6c-openshift-default-dockercfg-7pkzn\""
Oct 30 20:02:34.286634 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.286598    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2911d92-5384-4bbf-b875-c846a7df7459" path="/var/lib/kubelet/pods/c2911d92-5384-4bbf-b875-c846a7df7459/volumes"
Oct 30 20:02:34.287369 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.287346    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7558ea4-5322-443f-9847-857688238559" path="/var/lib/kubelet/pods/c7558ea4-5322-443f-9847-857688238559/volumes"
Oct 30 20:02:34.316671 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.316642    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c51f58-c2b5-4cef-858e-512b42dba670-istiod-ca-cert\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.316831 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.316790    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-istio-data\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.316956 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.316926    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c51f58-c2b5-4cef-858e-512b42dba670-istio-token\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.317034 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.317011    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-workload-socket\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.317096 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.317078    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-istio-envoy\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.317197 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.317177    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6fs7\" (UniqueName: \"kubernetes.io/projected/04c51f58-c2b5-4cef-858e-512b42dba670-kube-api-access-p6fs7\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.317272 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.317258    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c51f58-c2b5-4cef-858e-512b42dba670-istio-podinfo\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.317433 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.317417    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-credential-socket\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.317606 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.317591    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-workload-certs\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.436884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.431420    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-workload-certs\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.436884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.431530    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c51f58-c2b5-4cef-858e-512b42dba670-istiod-ca-cert\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.436884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.431653    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-istio-data\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.436884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.431690    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c51f58-c2b5-4cef-858e-512b42dba670-istio-token\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.436884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.431726    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-workload-socket\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.436884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.431784    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-istio-envoy\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.436884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.431854    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fs7\" (UniqueName: \"kubernetes.io/projected/04c51f58-c2b5-4cef-858e-512b42dba670-kube-api-access-p6fs7\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.436884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.431967    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c51f58-c2b5-4cef-858e-512b42dba670-istio-podinfo\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.436884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.432002    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-credential-socket\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.436884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.432536    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-credential-socket\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.436884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.432751    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-workload-certs\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.436884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.433360    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c51f58-c2b5-4cef-858e-512b42dba670-istiod-ca-cert\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.436884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.433562    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-istio-data\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.473092 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.464217    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-workload-socket\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.477522 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.477439    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c51f58-c2b5-4cef-858e-512b42dba670-istio-podinfo\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.499887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.499176    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-istio-envoy\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.499887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.499684    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c51f58-c2b5-4cef-858e-512b42dba670-istio-token\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.566994 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.547945    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6fs7\" (UniqueName: \"kubernetes.io/projected/04c51f58-c2b5-4cef-858e-512b42dba670-kube-api-access-p6fs7\") pod \"e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") " pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.566994 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.551724    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:34.607170 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:34.599087    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:02:35.199656 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:02:35.199624    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c51f58_c2b5_4cef_858e_512b42dba670.slice/crio-a06a0b85163faa0f3f8596d6386b0485f68f8f35793c146e2673a3c9340da913 WatchSource:0}: Error finding container a06a0b85163faa0f3f8596d6386b0485f68f8f35793c146e2673a3c9340da913: Status 404 returned error can't find the container with id a06a0b85163faa0f3f8596d6386b0485f68f8f35793c146e2673a3c9340da913
Oct 30 20:02:35.200933 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:35.200907    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"]
Oct 30 20:02:35.203362 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:35.203324    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:02:35.203450 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:35.203419    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:02:35.203501 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:35.203453    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:02:35.979933 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:35.978806    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq" event={"ID":"04c51f58-c2b5-4cef-858e-512b42dba670","Type":"ContainerStarted","Data":"170d06bcbe3fa01a2f5ba04610511898c74a189174ec323cbbfb5e15f5f3eb49"}
Oct 30 20:02:35.979933 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:35.978914    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq" event={"ID":"04c51f58-c2b5-4cef-858e-512b42dba670","Type":"ContainerStarted","Data":"a06a0b85163faa0f3f8596d6386b0485f68f8f35793c146e2673a3c9340da913"}
Oct 30 20:02:36.036006 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:36.035950    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq" podStartSLOduration=2.035935072 podStartE2EDuration="2.035935072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:36.02530666 +0000 UTC m=+1955.917920984" watchObservedRunningTime="2026-10-30 20:02:36.035935072 +0000 UTC m=+1955.928549395"
Oct 30 20:02:36.563978 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:36.563933    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:36.608455 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:36.608421    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:36.994061 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:36.994022    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:36.995526 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:36.995487    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:02:47.340694 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:47.340657    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"]
Oct 30 20:02:47.346959 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:47.346930    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"
Oct 30 20:02:47.357763 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:47.357742    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:02:47.357885 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:47.357808    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:02:47.430842 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:47.430808    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"]
Oct 30 20:02:47.514321 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:47.514274    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j\" (UID: \"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"
Oct 30 20:02:47.514503 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:47.514370    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88jq2\" (UniqueName: \"kubernetes.io/projected/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-kube-api-access-88jq2\") pod \"maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j\" (UID: \"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"
Oct 30 20:02:47.514503 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:47.514475    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j\" (UID: \"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"
Oct 30 20:02:47.616624 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:47.616597    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j\" (UID: \"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"
Oct 30 20:02:47.616828 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:47.616648    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88jq2\" (UniqueName: \"kubernetes.io/projected/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-kube-api-access-88jq2\") pod \"maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j\" (UID: \"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"
Oct 30 20:02:47.616828 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:47.616692    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j\" (UID: \"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"
Oct 30 20:02:47.617442 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:47.617410    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j\" (UID: \"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"
Oct 30 20:02:47.619393 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:47.619374    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j\" (UID: \"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"
Oct 30 20:02:47.634231 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:47.634207    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88jq2\" (UniqueName: \"kubernetes.io/projected/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-kube-api-access-88jq2\") pod \"maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j\" (UID: \"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"
Oct 30 20:02:47.663401 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:47.663354    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"
Oct 30 20:02:48.033753 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:48.033721    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"]
Oct 30 20:02:48.034218 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:02:48.034183    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d4e1a0_7a76_4dbb_80c5_a2903a5bd819.slice/crio-d28e62d843c182de6e33f24c94b8be6583bbf6ef51da36f5b9f107b898376613 WatchSource:0}: Error finding container d28e62d843c182de6e33f24c94b8be6583bbf6ef51da36f5b9f107b898376613: Status 404 returned error can't find the container with id d28e62d843c182de6e33f24c94b8be6583bbf6ef51da36f5b9f107b898376613
Oct 30 20:02:48.332414 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:48.332323    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j" event={"ID":"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819","Type":"ContainerStarted","Data":"d28e62d843c182de6e33f24c94b8be6583bbf6ef51da36f5b9f107b898376613"}
Oct 30 20:02:49.342753 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.342671    4823 generic.go:356] "Generic (PLEG): container finished" podID="d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819" containerID="8a6e7b3bafa9fd04d4a9503185f1196149094ca84348f93750bd5c9a7e7859d0" exitCode=0
Oct 30 20:02:49.343160 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.342759    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j" event={"ID":"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819","Type":"ContainerDied","Data":"8a6e7b3bafa9fd04d4a9503185f1196149094ca84348f93750bd5c9a7e7859d0"}
Oct 30 20:02:49.415682 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.415648    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"]
Oct 30 20:02:49.421154 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.421127    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:02:49.434888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.432219    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3a4f02-serving-cert\""
Oct 30 20:02:49.448935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.448856    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"]
Oct 30 20:02:49.591884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.565225    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/236fd49d-b067-40ba-83ab-def38b3b0862-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g\" (UID: \"236fd49d-b067-40ba-83ab-def38b3b0862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:02:49.595940 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.593700    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/236fd49d-b067-40ba-83ab-def38b3b0862-metrics-certs\") pod \"maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g\" (UID: \"236fd49d-b067-40ba-83ab-def38b3b0862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:02:49.595940 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.593912    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/236fd49d-b067-40ba-83ab-def38b3b0862-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g\" (UID: \"236fd49d-b067-40ba-83ab-def38b3b0862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:02:49.595940 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.594139    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftzxg\" (UniqueName: \"kubernetes.io/projected/236fd49d-b067-40ba-83ab-def38b3b0862-kube-api-access-ftzxg\") pod \"maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g\" (UID: \"236fd49d-b067-40ba-83ab-def38b3b0862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:02:49.695451 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.695412    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/236fd49d-b067-40ba-83ab-def38b3b0862-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g\" (UID: \"236fd49d-b067-40ba-83ab-def38b3b0862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:02:49.695451 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.695451    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/236fd49d-b067-40ba-83ab-def38b3b0862-metrics-certs\") pod \"maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g\" (UID: \"236fd49d-b067-40ba-83ab-def38b3b0862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:02:49.695727 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.695529    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/236fd49d-b067-40ba-83ab-def38b3b0862-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g\" (UID: \"236fd49d-b067-40ba-83ab-def38b3b0862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:02:49.695727 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.695591    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftzxg\" (UniqueName: \"kubernetes.io/projected/236fd49d-b067-40ba-83ab-def38b3b0862-kube-api-access-ftzxg\") pod \"maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g\" (UID: \"236fd49d-b067-40ba-83ab-def38b3b0862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:02:49.696317 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.696292    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/236fd49d-b067-40ba-83ab-def38b3b0862-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g\" (UID: \"236fd49d-b067-40ba-83ab-def38b3b0862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:02:49.698217 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.698197    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/236fd49d-b067-40ba-83ab-def38b3b0862-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g\" (UID: \"236fd49d-b067-40ba-83ab-def38b3b0862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:02:49.698317 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.698215    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/236fd49d-b067-40ba-83ab-def38b3b0862-metrics-certs\") pod \"maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g\" (UID: \"236fd49d-b067-40ba-83ab-def38b3b0862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:02:49.708529 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.708502    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftzxg\" (UniqueName: \"kubernetes.io/projected/236fd49d-b067-40ba-83ab-def38b3b0862-kube-api-access-ftzxg\") pod \"maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g\" (UID: \"236fd49d-b067-40ba-83ab-def38b3b0862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:02:49.755604 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:49.755575    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:02:50.071178 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:50.071143    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"]
Oct 30 20:02:50.071987 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:02:50.071960    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236fd49d_b067_40ba_83ab_def38b3b0862.slice/crio-90449f36c9613877ad139831d2c00abe95de0e0aac365d8e3b5e1e5ab6c46bec WatchSource:0}: Error finding container 90449f36c9613877ad139831d2c00abe95de0e0aac365d8e3b5e1e5ab6c46bec: Status 404 returned error can't find the container with id 90449f36c9613877ad139831d2c00abe95de0e0aac365d8e3b5e1e5ab6c46bec
Oct 30 20:02:50.497886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:50.497459    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g" event={"ID":"236fd49d-b067-40ba-83ab-def38b3b0862","Type":"ContainerStarted","Data":"90449f36c9613877ad139831d2c00abe95de0e0aac365d8e3b5e1e5ab6c46bec"}
Oct 30 20:02:50.693236 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:50.693209    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"
Oct 30 20:02:50.795521 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:50.795437    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-maas-api-xks-ca\") pod \"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819\" (UID: \"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819\") "
Oct 30 20:02:50.795696 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:50.795579    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-maas-api-openshift-ca\") pod \"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819\" (UID: \"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819\") "
Oct 30 20:02:50.795696 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:50.795642    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-kube-api-access-88jq2\" (UniqueName: \"kubernetes.io/projected/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-kube-api-access-88jq2\") pod \"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819\" (UID: \"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819\") "
Oct 30 20:02:50.796042 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:50.796013    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-maas-api-openshift-ca" pod "d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819" (UID: "d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:50.797209 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:50.797181    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:50.798332 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:50.798308    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-kube-api-access-88jq2" pod "d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819" (UID: "d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819"). InnerVolumeSpecName "kube-api-access-88jq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:50.798916 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:50.798848    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-maas-api-xks-ca" pod "d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819" (UID: "d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:50.914341 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:50.913219    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:50.914341 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:50.913258    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88jq2\" (UniqueName: \"kubernetes.io/projected/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819-kube-api-access-88jq2\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:51.529644 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:51.529612    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"
Oct 30 20:02:51.530101 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:51.529653    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j" event={"ID":"d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819","Type":"ContainerDied","Data":"d28e62d843c182de6e33f24c94b8be6583bbf6ef51da36f5b9f107b898376613"}
Oct 30 20:02:51.530101 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:51.529701    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d28e62d843c182de6e33f24c94b8be6583bbf6ef51da36f5b9f107b898376613"
Oct 30 20:02:51.531879 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:51.531829    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g" event={"ID":"236fd49d-b067-40ba-83ab-def38b3b0862","Type":"ContainerStarted","Data":"284f88e9291f588bfdf20626fab8ca83026f0c6b93adeefd874a05ace3ebcdc2"}
Oct 30 20:02:51.532007 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:51.531904    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:02:51.552940 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:51.552895    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g" podStartSLOduration=2.080104352 podStartE2EDuration="2.552878706s" totalImagesPullingTime="472.774354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:51.550901707 +0000 UTC m=+1971.443516028" watchObservedRunningTime="2026-10-30 20:02:51.552878706 +0000 UTC m=+1971.445493028"
Oct 30 20:02:56.720439 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:56.720406    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"]
Oct 30 20:02:56.724870 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:56.724840    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c50e1e-rwvcl"]
Oct 30 20:02:57.552275 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:57.552247    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:02:58.281559 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:02:58.281520    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6654a377-ad63-4c0a-bf8a-d358e3f56ad4" path="/var/lib/kubelet/pods/6654a377-ad63-4c0a-bf8a-d358e3f56ad4/volumes"
Oct 30 20:03:02.004493 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:02.004462    4823 scope.go:122] "RemoveContainer" containerID="06366f3a1904b84da24a7ed55755962f04796df16bab68bd7a93296fe3fee2d4"
Oct 30 20:03:02.016113 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:03:02.016080    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:03:02.016776 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:02.016761    4823 scope.go:122] "RemoveContainer" containerID="71f947678c3067af7b54099edaad79c2492caa65ddb65db205e7ba17317d34b6"
Oct 30 20:03:02.026959 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:02.026936    4823 scope.go:122] "RemoveContainer" containerID="5b969b5f79707c83e4b70ddcc05702c3224fb7444270c4176f2068b68121c061"
Oct 30 20:03:06.854896 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:06.854843    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"]
Oct 30 20:03:06.855418 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:06.855120    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2" podUID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerName="istio-proxy" containerID="cri-o://e63b55df1ce8933b27855603072ce302aff50b293456e330fd14f583d347d982" gracePeriod=30
Oct 30 20:03:07.399960 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.399924    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"]
Oct 30 20:03:07.400805 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.400773    4823 cpu_manager.go:414] "containerMap: removing container" podUID="d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819" containerName="revoke-keys"
Oct 30 20:03:07.400805 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.400797    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819" containerName="revoke-keys"
Oct 30 20:03:07.401102 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.401084    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819" containerName="revoke-keys"
Oct 30 20:03:07.407119 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.407099    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"
Oct 30 20:03:07.415542 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.415503    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:03:07.415780 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.415607    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:03:07.419651 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.419628    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"]
Oct 30 20:03:07.503051 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.503011    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fc44\" (UniqueName: \"kubernetes.io/projected/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-kube-api-access-5fc44\") pod \"maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x\" (UID: \"28617b93-ee6f-4ec6-a3f1-e38018ea27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"
Oct 30 20:03:07.503226 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.503081    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x\" (UID: \"28617b93-ee6f-4ec6-a3f1-e38018ea27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"
Oct 30 20:03:07.503226 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.503208    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x\" (UID: \"28617b93-ee6f-4ec6-a3f1-e38018ea27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"
Oct 30 20:03:07.604966 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.604928    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fc44\" (UniqueName: \"kubernetes.io/projected/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-kube-api-access-5fc44\") pod \"maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x\" (UID: \"28617b93-ee6f-4ec6-a3f1-e38018ea27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"
Oct 30 20:03:07.605161 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.604996    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x\" (UID: \"28617b93-ee6f-4ec6-a3f1-e38018ea27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"
Oct 30 20:03:07.605161 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.605091    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x\" (UID: \"28617b93-ee6f-4ec6-a3f1-e38018ea27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"
Oct 30 20:03:07.605722 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.605695    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x\" (UID: \"28617b93-ee6f-4ec6-a3f1-e38018ea27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"
Oct 30 20:03:07.607678 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.607658    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x\" (UID: \"28617b93-ee6f-4ec6-a3f1-e38018ea27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"
Oct 30 20:03:07.620944 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.620923    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fc44\" (UniqueName: \"kubernetes.io/projected/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-kube-api-access-5fc44\") pod \"maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x\" (UID: \"28617b93-ee6f-4ec6-a3f1-e38018ea27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"
Oct 30 20:03:07.732464 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:07.732379    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"
Oct 30 20:03:08.288245 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:08.288206    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"]
Oct 30 20:03:08.298199 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:03:08.298162    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28617b93_ee6f_4ec6_a3f1_e38018ea27e8.slice/crio-09ace902c4abd506b4315e4eb5f348e90c10ff6d82fc6c18e4cfac2550eeee1f WatchSource:0}: Error finding container 09ace902c4abd506b4315e4eb5f348e90c10ff6d82fc6c18e4cfac2550eeee1f: Status 404 returned error can't find the container with id 09ace902c4abd506b4315e4eb5f348e90c10ff6d82fc6c18e4cfac2550eeee1f
Oct 30 20:03:09.038544 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:09.038443    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x" event={"ID":"28617b93-ee6f-4ec6-a3f1-e38018ea27e8","Type":"ContainerStarted","Data":"3a7e3009d2ffcb12d308c407f743e5fb005fc2305471e10d10cbb285bbe44d92"}
Oct 30 20:03:09.038544 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:09.038501    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x" event={"ID":"28617b93-ee6f-4ec6-a3f1-e38018ea27e8","Type":"ContainerStarted","Data":"09ace902c4abd506b4315e4eb5f348e90c10ff6d82fc6c18e4cfac2550eeee1f"}
Oct 30 20:03:09.059767 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:09.059716    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x" podStartSLOduration=2.059703238 podStartE2EDuration="2.059703238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:09.056146902 +0000 UTC m=+1988.948761227" watchObservedRunningTime="2026-10-30 20:03:09.059703238 +0000 UTC m=+1988.952317561"
Oct 30 20:03:10.065950 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:10.065909    4823 generic.go:356] "Generic (PLEG): container finished" podID="28617b93-ee6f-4ec6-a3f1-e38018ea27e8" containerID="3a7e3009d2ffcb12d308c407f743e5fb005fc2305471e10d10cbb285bbe44d92" exitCode=0
Oct 30 20:03:10.066450 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:10.065998    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x" event={"ID":"28617b93-ee6f-4ec6-a3f1-e38018ea27e8","Type":"ContainerDied","Data":"3a7e3009d2ffcb12d308c407f743e5fb005fc2305471e10d10cbb285bbe44d92"}
Oct 30 20:03:11.750361 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:11.750335    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"
Oct 30 20:03:11.858329 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:11.858296    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-maas-api-xks-ca\") pod \"28617b93-ee6f-4ec6-a3f1-e38018ea27e8\" (UID: \"28617b93-ee6f-4ec6-a3f1-e38018ea27e8\") "
Oct 30 20:03:11.858539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:11.858380    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-maas-api-openshift-ca\") pod \"28617b93-ee6f-4ec6-a3f1-e38018ea27e8\" (UID: \"28617b93-ee6f-4ec6-a3f1-e38018ea27e8\") "
Oct 30 20:03:11.858741 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:11.858716    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-maas-api-openshift-ca" pod "28617b93-ee6f-4ec6-a3f1-e38018ea27e8" (UID: "28617b93-ee6f-4ec6-a3f1-e38018ea27e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:11.858816 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:11.858759    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-kube-api-access-5fc44\" (UniqueName: \"kubernetes.io/projected/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-kube-api-access-5fc44\") pod \"28617b93-ee6f-4ec6-a3f1-e38018ea27e8\" (UID: \"28617b93-ee6f-4ec6-a3f1-e38018ea27e8\") "
Oct 30 20:03:11.859602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:11.859576    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:11.862851 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:11.862818    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-maas-api-xks-ca" pod "28617b93-ee6f-4ec6-a3f1-e38018ea27e8" (UID: "28617b93-ee6f-4ec6-a3f1-e38018ea27e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:11.863373 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:11.863351    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-kube-api-access-5fc44" pod "28617b93-ee6f-4ec6-a3f1-e38018ea27e8" (UID: "28617b93-ee6f-4ec6-a3f1-e38018ea27e8"). InnerVolumeSpecName "kube-api-access-5fc44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:11.982886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:11.969846    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fc44\" (UniqueName: \"kubernetes.io/projected/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-kube-api-access-5fc44\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:11.982886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:11.969906    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28617b93-ee6f-4ec6-a3f1-e38018ea27e8-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.163823 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.163733    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"
Oct 30 20:03:12.163823 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.163761    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x" event={"ID":"28617b93-ee6f-4ec6-a3f1-e38018ea27e8","Type":"ContainerDied","Data":"09ace902c4abd506b4315e4eb5f348e90c10ff6d82fc6c18e4cfac2550eeee1f"}
Oct 30 20:03:12.163823 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.163815    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09ace902c4abd506b4315e4eb5f348e90c10ff6d82fc6c18e4cfac2550eeee1f"
Oct 30 20:03:12.169262 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.168940    4823 generic.go:356] "Generic (PLEG): container finished" podID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerID="e63b55df1ce8933b27855603072ce302aff50b293456e330fd14f583d347d982" exitCode=0
Oct 30 20:03:12.170217 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.169546    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2" event={"ID":"7b162ce0-cc0c-4f76-802d-9175c518f6df","Type":"ContainerDied","Data":"e63b55df1ce8933b27855603072ce302aff50b293456e330fd14f583d347d982"}
Oct 30 20:03:12.170217 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.169616    4823 scope.go:122] "RemoveContainer" containerID="b8ed26a374750bd61410b7fcf446c3189fe44053e8a8cc56e1a1ea9565a260b0"
Oct 30 20:03:12.240681 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.240656    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:03:12.379922 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.379889    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-credential-socket\") pod \"7b162ce0-cc0c-4f76-802d-9175c518f6df\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") "
Oct 30 20:03:12.380116 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.379946    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b162ce0-cc0c-4f76-802d-9175c518f6df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b162ce0-cc0c-4f76-802d-9175c518f6df-istiod-ca-cert\") pod \"7b162ce0-cc0c-4f76-802d-9175c518f6df\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") "
Oct 30 20:03:12.380116 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.380048    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-workload-certs\") pod \"7b162ce0-cc0c-4f76-802d-9175c518f6df\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") "
Oct 30 20:03:12.380233 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.380122    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-workload-socket\") pod \"7b162ce0-cc0c-4f76-802d-9175c518f6df\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") "
Oct 30 20:03:12.380233 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.380150    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b162ce0-cc0c-4f76-802d-9175c518f6df-kube-api-access-8fsdn\" (UniqueName: \"kubernetes.io/projected/7b162ce0-cc0c-4f76-802d-9175c518f6df-kube-api-access-8fsdn\") pod \"7b162ce0-cc0c-4f76-802d-9175c518f6df\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") "
Oct 30 20:03:12.380335 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.380256    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-podinfo\") pod \"7b162ce0-cc0c-4f76-802d-9175c518f6df\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") "
Oct 30 20:03:12.380335 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.380286    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-envoy\") pod \"7b162ce0-cc0c-4f76-802d-9175c518f6df\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") "
Oct 30 20:03:12.380440 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.380397    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-workload-socket" pod "7b162ce0-cc0c-4f76-802d-9175c518f6df" (UID: "7b162ce0-cc0c-4f76-802d-9175c518f6df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:12.380440 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.380414    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-workload-certs" pod "7b162ce0-cc0c-4f76-802d-9175c518f6df" (UID: "7b162ce0-cc0c-4f76-802d-9175c518f6df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:12.380551 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.380421    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b162ce0-cc0c-4f76-802d-9175c518f6df-istiod-ca-cert" pod "7b162ce0-cc0c-4f76-802d-9175c518f6df" (UID: "7b162ce0-cc0c-4f76-802d-9175c518f6df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:12.380551 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.380421    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-data\") pod \"7b162ce0-cc0c-4f76-802d-9175c518f6df\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") "
Oct 30 20:03:12.380652 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.380588    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-token\" (UniqueName: \"kubernetes.io/projected/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-token\") pod \"7b162ce0-cc0c-4f76-802d-9175c518f6df\" (UID: \"7b162ce0-cc0c-4f76-802d-9175c518f6df\") "
Oct 30 20:03:12.380652 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.380618    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-data" pod "7b162ce0-cc0c-4f76-802d-9175c518f6df" (UID: "7b162ce0-cc0c-4f76-802d-9175c518f6df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:12.380652 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.380211    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-credential-socket" pod "7b162ce0-cc0c-4f76-802d-9175c518f6df" (UID: "7b162ce0-cc0c-4f76-802d-9175c518f6df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:12.382346 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.382318    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.382573 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.382457    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b162ce0-cc0c-4f76-802d-9175c518f6df-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.382573 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.382482    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.382573 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.382524    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.382573 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.382542    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.383563 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.383031    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b162ce0-cc0c-4f76-802d-9175c518f6df-kube-api-access-8fsdn" pod "7b162ce0-cc0c-4f76-802d-9175c518f6df" (UID: "7b162ce0-cc0c-4f76-802d-9175c518f6df"). InnerVolumeSpecName "kube-api-access-8fsdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:12.383563 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.383058    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-podinfo" pod "7b162ce0-cc0c-4f76-802d-9175c518f6df" (UID: "7b162ce0-cc0c-4f76-802d-9175c518f6df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:03:12.383563 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.383229    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-token" pod "7b162ce0-cc0c-4f76-802d-9175c518f6df" (UID: "7b162ce0-cc0c-4f76-802d-9175c518f6df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:12.383767 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.383749    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-envoy" pod "7b162ce0-cc0c-4f76-802d-9175c518f6df" (UID: "7b162ce0-cc0c-4f76-802d-9175c518f6df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:03:12.484477 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.484388    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fsdn\" (UniqueName: \"kubernetes.io/projected/7b162ce0-cc0c-4f76-802d-9175c518f6df-kube-api-access-8fsdn\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.484477 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.484422    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.484477 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.484435    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:12.484477 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:12.484451    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b162ce0-cc0c-4f76-802d-9175c518f6df-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:13.181927 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:13.181895    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"
Oct 30 20:03:13.181927 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:13.181901    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2" event={"ID":"7b162ce0-cc0c-4f76-802d-9175c518f6df","Type":"ContainerDied","Data":"f4da11d6b23f11b9455710125768a993a1e53c9f7cc55b83dd1e970b8d4e811f"}
Oct 30 20:03:13.182448 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:13.181985    4823 scope.go:122] "RemoveContainer" containerID="e63b55df1ce8933b27855603072ce302aff50b293456e330fd14f583d347d982"
Oct 30 20:03:13.217533 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:13.217508    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"]
Oct 30 20:03:13.224761 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:13.224731    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-m7jq2"]
Oct 30 20:03:14.282236 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:14.282204    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b162ce0-cc0c-4f76-802d-9175c518f6df" path="/var/lib/kubelet/pods/7b162ce0-cc0c-4f76-802d-9175c518f6df/volumes"
Oct 30 20:03:19.112654 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:19.112617    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:03:25.291054 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.291016    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"]
Oct 30 20:03:25.292795 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.292764    4823 cpu_manager.go:414] "containerMap: removing container" podUID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerName="istio-proxy"
Oct 30 20:03:25.292975 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.292959    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerName="istio-proxy"
Oct 30 20:03:25.293154 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.293139    4823 cpu_manager.go:414] "containerMap: removing container" podUID="28617b93-ee6f-4ec6-a3f1-e38018ea27e8" containerName="revoke-keys"
Oct 30 20:03:25.293255 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.293243    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28617b93-ee6f-4ec6-a3f1-e38018ea27e8" containerName="revoke-keys"
Oct 30 20:03:25.293656 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.293639    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="28617b93-ee6f-4ec6-a3f1-e38018ea27e8" containerName="revoke-keys"
Oct 30 20:03:25.293817 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.293804    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerName="istio-proxy"
Oct 30 20:03:25.294002 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.293987    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerName="istio-proxy"
Oct 30 20:03:25.299038 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.299016    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:25.309815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.309780    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"]
Oct 30 20:03:25.313533 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.310895    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-db02a7-serving-cert\""
Oct 30 20:03:25.330042 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.330009    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-metrics-certs\") pod \"maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:25.330296 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.330273    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlftl\" (UniqueName: \"kubernetes.io/projected/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-kube-api-access-hlftl\") pod \"maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:25.330393 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.330377    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:25.330659 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.330592    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-maas-api-tls\") pod \"maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:25.482786 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.479093    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlftl\" (UniqueName: \"kubernetes.io/projected/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-kube-api-access-hlftl\") pod \"maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:25.491088 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.491046    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:25.514511 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.514469    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-maas-api-tls\") pod \"maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:25.523407 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.523367    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-metrics-certs\") pod \"maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:25.554673 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:03:25.516252    4823 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-serving-cert: secret "maas-api-e2e-inf-b-db02a7-serving-cert" not found
Oct 30 20:03:25.555026 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:03:25.555007    4823 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-maas-api-tls podName:96d2936e-4782-4bc3-b4a3-2ebb19701fdc nodeName:}" failed. No retries permitted until 2026-10-30 20:03:26.054980589 +0000 UTC m=+2005.947594894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-maas-api-tls") pod "maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk" (UID: "96d2936e-4782-4bc3-b4a3-2ebb19701fdc") : secret "maas-api-e2e-inf-b-db02a7-serving-cert" not found
Oct 30 20:03:25.555427 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.492820    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:25.578893 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.572399    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlftl\" (UniqueName: \"kubernetes.io/projected/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-kube-api-access-hlftl\") pod \"maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:25.582888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:25.581500    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-metrics-certs\") pod \"maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:26.100883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:26.100272    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-maas-api-tls\") pod \"maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:26.133605 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:26.133568    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-maas-api-tls\") pod \"maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:26.230882 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:26.230709    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:26.582836 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:03:26.582791    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d2936e_4782_4bc3_b4a3_2ebb19701fdc.slice/crio-d78d3a9e1ecdf2b02554ddc50db91eb83592ed574a469b4ff4ad384e63204528 WatchSource:0}: Error finding container d78d3a9e1ecdf2b02554ddc50db91eb83592ed574a469b4ff4ad384e63204528: Status 404 returned error can't find the container with id d78d3a9e1ecdf2b02554ddc50db91eb83592ed574a469b4ff4ad384e63204528
Oct 30 20:03:26.583197 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:26.582883    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"]
Oct 30 20:03:26.655899 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:26.649360    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk" event={"ID":"96d2936e-4782-4bc3-b4a3-2ebb19701fdc","Type":"ContainerStarted","Data":"d78d3a9e1ecdf2b02554ddc50db91eb83592ed574a469b4ff4ad384e63204528"}
Oct 30 20:03:27.669275 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:27.669241    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk" event={"ID":"96d2936e-4782-4bc3-b4a3-2ebb19701fdc","Type":"ContainerStarted","Data":"5ae5f60038a8a56c9d2be2b90dc60ee3951d36ed938fc3c2fe3fd31bc6678343"}
Oct 30 20:03:27.669794 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:27.669339    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:27.693990 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:27.693939    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk" podStartSLOduration=2.256636161 podStartE2EDuration="2.69392565s" totalImagesPullingTime="437.289489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:27.689146558 +0000 UTC m=+2007.581760882" watchObservedRunningTime="2026-10-30 20:03:27.69392565 +0000 UTC m=+2007.586539973"
Oct 30 20:03:29.791700 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:29.791666    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"]
Oct 30 20:03:29.796304 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:29.796274    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ed8b50-xwg2j"]
Oct 30 20:03:30.281831 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.281793    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819" path="/var/lib/kubelet/pods/d9d4e1a0-7a76-4dbb-80c5-a2903a5bd819/volumes"
Oct 30 20:03:30.727518 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.727487    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"]
Oct 30 20:03:30.728326 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.728310    4823 cpu_manager.go:414] "containerMap: removing container" podUID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerName="istio-proxy"
Oct 30 20:03:30.728380 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.728328    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b162ce0-cc0c-4f76-802d-9175c518f6df" containerName="istio-proxy"
Oct 30 20:03:30.733793 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.733774    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"
Oct 30 20:03:30.746030 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.740040    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:03:30.746030 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.740155    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:03:30.746776 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.746756    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"]
Oct 30 20:03:30.824892 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.820347    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfb5d\" (UniqueName: \"kubernetes.io/projected/08bc865f-a124-4e5d-9cdc-a296aac31513-kube-api-access-cfb5d\") pod \"maas-api-revoke-sub-660ac090-9ssrt\" (UID: \"08bc865f-a124-4e5d-9cdc-a296aac31513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"
Oct 30 20:03:30.824892 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.820681    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08bc865f-a124-4e5d-9cdc-a296aac31513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-660ac090-9ssrt\" (UID: \"08bc865f-a124-4e5d-9cdc-a296aac31513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"
Oct 30 20:03:30.849115 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.832425    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08bc865f-a124-4e5d-9cdc-a296aac31513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-660ac090-9ssrt\" (UID: \"08bc865f-a124-4e5d-9cdc-a296aac31513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"
Oct 30 20:03:30.935782 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.934917    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08bc865f-a124-4e5d-9cdc-a296aac31513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-660ac090-9ssrt\" (UID: \"08bc865f-a124-4e5d-9cdc-a296aac31513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"
Oct 30 20:03:30.936023 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.935813    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08bc865f-a124-4e5d-9cdc-a296aac31513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-660ac090-9ssrt\" (UID: \"08bc865f-a124-4e5d-9cdc-a296aac31513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"
Oct 30 20:03:30.936023 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.935931    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08bc865f-a124-4e5d-9cdc-a296aac31513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-660ac090-9ssrt\" (UID: \"08bc865f-a124-4e5d-9cdc-a296aac31513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"
Oct 30 20:03:30.936155 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.936061    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfb5d\" (UniqueName: \"kubernetes.io/projected/08bc865f-a124-4e5d-9cdc-a296aac31513-kube-api-access-cfb5d\") pod \"maas-api-revoke-sub-660ac090-9ssrt\" (UID: \"08bc865f-a124-4e5d-9cdc-a296aac31513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"
Oct 30 20:03:30.938820 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.938781    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08bc865f-a124-4e5d-9cdc-a296aac31513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-660ac090-9ssrt\" (UID: \"08bc865f-a124-4e5d-9cdc-a296aac31513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"
Oct 30 20:03:30.949272 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:30.949243    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfb5d\" (UniqueName: \"kubernetes.io/projected/08bc865f-a124-4e5d-9cdc-a296aac31513-kube-api-access-cfb5d\") pod \"maas-api-revoke-sub-660ac090-9ssrt\" (UID: \"08bc865f-a124-4e5d-9cdc-a296aac31513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"
Oct 30 20:03:31.065890 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:31.065793    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"
Oct 30 20:03:31.375543 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:31.375516    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"]
Oct 30 20:03:31.376888 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:03:31.376849    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bc865f_a124_4e5d_9cdc_a296aac31513.slice/crio-33e5ddd9998335ecff4f4a5b0cfda3c15951247f79033445336b5ccd23a92283 WatchSource:0}: Error finding container 33e5ddd9998335ecff4f4a5b0cfda3c15951247f79033445336b5ccd23a92283: Status 404 returned error can't find the container with id 33e5ddd9998335ecff4f4a5b0cfda3c15951247f79033445336b5ccd23a92283
Oct 30 20:03:31.748365 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:31.748285    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt" event={"ID":"08bc865f-a124-4e5d-9cdc-a296aac31513","Type":"ContainerStarted","Data":"33e5ddd9998335ecff4f4a5b0cfda3c15951247f79033445336b5ccd23a92283"}
Oct 30 20:03:32.017537 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:03:32.017442    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:03:32.759204 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:32.759118    4823 generic.go:356] "Generic (PLEG): container finished" podID="08bc865f-a124-4e5d-9cdc-a296aac31513" containerID="204ae4e2c4de9294ea9160ad2711c9f4a70610f14c449a6392ce6f5124b43769" exitCode=0
Oct 30 20:03:32.759204 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:32.759157    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt" event={"ID":"08bc865f-a124-4e5d-9cdc-a296aac31513","Type":"ContainerDied","Data":"204ae4e2c4de9294ea9160ad2711c9f4a70610f14c449a6392ce6f5124b43769"}
Oct 30 20:03:32.761056 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:32.761031    4823 generic.go:356] "Generic (PLEG): container finished" podID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" containerID="895393ad8bd262e70e9d798e3795bb459ae01e232eee17b1cc38488951747dcb" exitCode=0
Oct 30 20:03:32.761193 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:32.761104    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n" event={"ID":"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3","Type":"ContainerDied","Data":"895393ad8bd262e70e9d798e3795bb459ae01e232eee17b1cc38488951747dcb"}
Oct 30 20:03:32.761575 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:32.761559    4823 scope.go:122] "RemoveContainer" containerID="895393ad8bd262e70e9d798e3795bb459ae01e232eee17b1cc38488951747dcb"
Oct 30 20:03:32.763626 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:32.763601    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:03:32.763688 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:32.763670    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:03:32.763726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:32.763702    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:03:33.687474 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:33.687442    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:03:33.797958 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:33.797899    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n" event={"ID":"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3","Type":"ContainerStarted","Data":"d1c4a1a14dd539165306051c8cc55505d2ebcbefb42077eaefc0d04215f3da3e"}
Oct 30 20:03:34.145738 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:34.145715    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"
Oct 30 20:03:34.228435 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:34.228402    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08bc865f-a124-4e5d-9cdc-a296aac31513-kube-api-access-cfb5d\" (UniqueName: \"kubernetes.io/projected/08bc865f-a124-4e5d-9cdc-a296aac31513-kube-api-access-cfb5d\") pod \"08bc865f-a124-4e5d-9cdc-a296aac31513\" (UID: \"08bc865f-a124-4e5d-9cdc-a296aac31513\") "
Oct 30 20:03:34.228629 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:34.228599    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08bc865f-a124-4e5d-9cdc-a296aac31513-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08bc865f-a124-4e5d-9cdc-a296aac31513-maas-api-xks-ca\") pod \"08bc865f-a124-4e5d-9cdc-a296aac31513\" (UID: \"08bc865f-a124-4e5d-9cdc-a296aac31513\") "
Oct 30 20:03:34.228704 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:34.228659    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08bc865f-a124-4e5d-9cdc-a296aac31513-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08bc865f-a124-4e5d-9cdc-a296aac31513-maas-api-openshift-ca\") pod \"08bc865f-a124-4e5d-9cdc-a296aac31513\" (UID: \"08bc865f-a124-4e5d-9cdc-a296aac31513\") "
Oct 30 20:03:34.229158 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:34.229121    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08bc865f-a124-4e5d-9cdc-a296aac31513-maas-api-openshift-ca" pod "08bc865f-a124-4e5d-9cdc-a296aac31513" (UID: "08bc865f-a124-4e5d-9cdc-a296aac31513"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:34.229608 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:34.229582    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08bc865f-a124-4e5d-9cdc-a296aac31513-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:34.231472 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:34.231435    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08bc865f-a124-4e5d-9cdc-a296aac31513-kube-api-access-cfb5d" pod "08bc865f-a124-4e5d-9cdc-a296aac31513" (UID: "08bc865f-a124-4e5d-9cdc-a296aac31513"). InnerVolumeSpecName "kube-api-access-cfb5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:34.231625 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:34.231597    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08bc865f-a124-4e5d-9cdc-a296aac31513-maas-api-xks-ca" pod "08bc865f-a124-4e5d-9cdc-a296aac31513" (UID: "08bc865f-a124-4e5d-9cdc-a296aac31513"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:34.331672 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:34.331640    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfb5d\" (UniqueName: \"kubernetes.io/projected/08bc865f-a124-4e5d-9cdc-a296aac31513-kube-api-access-cfb5d\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:34.331672 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:34.331669    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08bc865f-a124-4e5d-9cdc-a296aac31513-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:34.645339 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:34.645310    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:03:34.646782 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:34.646756    4823 patch_prober.go:29] interesting pod/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 20:03:34.646926 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:34.646814    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n" podUID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 20:03:34.811966 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:34.811934    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"
Oct 30 20:03:34.811966 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:34.811950    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt" event={"ID":"08bc865f-a124-4e5d-9cdc-a296aac31513","Type":"ContainerDied","Data":"33e5ddd9998335ecff4f4a5b0cfda3c15951247f79033445336b5ccd23a92283"}
Oct 30 20:03:34.812383 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:34.812006    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33e5ddd9998335ecff4f4a5b0cfda3c15951247f79033445336b5ccd23a92283"
Oct 30 20:03:35.645239 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:35.645203    4823 patch_prober.go:29] interesting pod/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 20:03:35.645446 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:35.645292    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n" podUID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 20:03:36.646215 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:36.646177    4823 patch_prober.go:29] interesting pod/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 20:03:36.646797 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:36.646762    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n" podUID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 20:03:37.645186 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:37.645144    4823 patch_prober.go:29] interesting pod/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 20:03:37.645400 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:37.645211    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n" podUID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 20:03:37.990151 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:37.990039    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:03:38.656886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:38.655646    4823 patch_prober.go:29] interesting pod/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 20:03:38.656886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:38.655722    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n" podUID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 20:03:39.666886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:39.661598    4823 patch_prober.go:29] interesting pod/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 30 20:03:39.666886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:39.661667    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n" podUID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 30 20:03:40.650343 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:40.650255    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:03:40.651589 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:40.651567    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:03:41.108825 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.108786    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"]
Oct 30 20:03:41.110169 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.110134    4823 cpu_manager.go:414] "containerMap: removing container" podUID="08bc865f-a124-4e5d-9cdc-a296aac31513" containerName="revoke-keys"
Oct 30 20:03:41.110169 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.110168    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08bc865f-a124-4e5d-9cdc-a296aac31513" containerName="revoke-keys"
Oct 30 20:03:41.110420 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.110402    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="08bc865f-a124-4e5d-9cdc-a296aac31513" containerName="revoke-keys"
Oct 30 20:03:41.114713 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.114687    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.122608 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.122587    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-925e04-openshift-default-dockercfg-qwbbk\""
Oct 30 20:03:41.144824 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.144787    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"]
Oct 30 20:03:41.166992 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.166960    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835abdec-0719-4f64-b7e8-4e336b6f12bf-istiod-ca-cert\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.167171 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.167007    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-data\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.167171 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.167043    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-token\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.167264 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.167205    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-podinfo\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.167318 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.167270    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-credential-socket\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.167458 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.167439    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-workload-socket\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.167601 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.167584    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k47j\" (UniqueName: \"kubernetes.io/projected/835abdec-0719-4f64-b7e8-4e336b6f12bf-kube-api-access-8k47j\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.167682 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.167635    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-workload-certs\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.167751 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.167723    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-envoy\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.314897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.313775    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k47j\" (UniqueName: \"kubernetes.io/projected/835abdec-0719-4f64-b7e8-4e336b6f12bf-kube-api-access-8k47j\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.314897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.313852    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-workload-certs\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.314897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.313904    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-envoy\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.314897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.313963    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835abdec-0719-4f64-b7e8-4e336b6f12bf-istiod-ca-cert\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.314897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.313991    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-data\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.314897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.314021    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-token\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.314897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.314083    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-podinfo\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.314897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.314139    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-credential-socket\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.314897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.314321    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-workload-socket\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.345885 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.322584    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-workload-certs\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.346144 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.322847    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-data\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.346269 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.342085    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-workload-socket\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.349883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.346635    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-credential-socket\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.385983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.350691    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835abdec-0719-4f64-b7e8-4e336b6f12bf-istiod-ca-cert\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.386253 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.376008    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-envoy\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.387484 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.387454    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-token\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.395957 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.388461    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-podinfo\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.411343 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.411308    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k47j\" (UniqueName: \"kubernetes.io/projected/835abdec-0719-4f64-b7e8-4e336b6f12bf-kube-api-access-8k47j\") pod \"e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") " pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.444444 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.444405    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:41.907975 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.907947    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"]
Oct 30 20:03:41.914981 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.914949    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:41.923814 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.923751    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f5eee-openshift-default-dockercfg-dwlpb\""
Oct 30 20:03:41.978569 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:41.978488    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"]
Oct 30 20:03:42.026025 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.025989    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"]
Oct 30 20:03:42.035940 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:03:42.035024    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835abdec_0719_4f64_b7e8_4e336b6f12bf.slice/crio-2e4d9fab465ebb8ef1892b393c97e71f94614ddc9d8b32ffe59dc1e2c0755aad WatchSource:0}: Error finding container 2e4d9fab465ebb8ef1892b393c97e71f94614ddc9d8b32ffe59dc1e2c0755aad: Status 404 returned error can't find the container with id 2e4d9fab465ebb8ef1892b393c97e71f94614ddc9d8b32ffe59dc1e2c0755aad
Oct 30 20:03:42.039693 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.039103    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:03:42.039693 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.039168    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:03:42.039693 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.039198    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:03:42.076614 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.076394    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-workload-socket\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.076614 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.076506    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-podinfo\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.076832 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.076721    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-workload-certs\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.076832 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.076763    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istiod-ca-cert\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.076832 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.076800    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mdtf\" (UniqueName: \"kubernetes.io/projected/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-kube-api-access-2mdtf\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.077039 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.077011    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-credential-socket\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.077097 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.077075    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-envoy\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.077155 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.077102    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-token\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.077155 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.077136    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-data\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.179407 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.179373    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-credential-socket\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.179913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.179450    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-envoy\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.179913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.179482    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-token\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.179913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.179508    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-data\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.179913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.179640    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-workload-socket\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.179913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.179728    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-podinfo\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.179913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.179849    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-workload-certs\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.179913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.179892    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-credential-socket\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.179913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.179905    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istiod-ca-cert\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.181946 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.180269    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-workload-certs\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.181946 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.180296    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-workload-socket\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.181946 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.180378    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mdtf\" (UniqueName: \"kubernetes.io/projected/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-kube-api-access-2mdtf\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.181946 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.181038    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istiod-ca-cert\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.182376 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.182351    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-data\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.183640 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.183487    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-envoy\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.185449 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.185401    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-podinfo\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.212922 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.209640    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-token\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.212922 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.212286    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mdtf\" (UniqueName: \"kubernetes.io/projected/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-kube-api-access-2mdtf\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:42.302936 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:42.298219    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:43.055804 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:43.055720    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj" event={"ID":"835abdec-0719-4f64-b7e8-4e336b6f12bf","Type":"ContainerStarted","Data":"b8cb4e161b5701d775fc397a00ff0cc8ed987ace4598c454879f20e69f1c0631"}
Oct 30 20:03:43.055804 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:43.055804    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj" event={"ID":"835abdec-0719-4f64-b7e8-4e336b6f12bf","Type":"ContainerStarted","Data":"2e4d9fab465ebb8ef1892b393c97e71f94614ddc9d8b32ffe59dc1e2c0755aad"}
Oct 30 20:03:43.081429 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:43.081399    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"]
Oct 30 20:03:43.081579 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:03:43.081408    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aaa5dc0_1ff0_4458_8181_9201b8dc5b15.slice/crio-b3bc9b939aec6cf61d331f64c325247570a012ba346f3e6b30e25850f5bb2ebc WatchSource:0}: Error finding container b3bc9b939aec6cf61d331f64c325247570a012ba346f3e6b30e25850f5bb2ebc: Status 404 returned error can't find the container with id b3bc9b939aec6cf61d331f64c325247570a012ba346f3e6b30e25850f5bb2ebc
Oct 30 20:03:43.084457 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:43.084423    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:03:43.084572 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:43.084486    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:03:43.084572 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:43.084516    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:03:43.094049 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:43.089984    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj" podStartSLOduration=2.089966477 podStartE2EDuration="2.089966477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:43.08746667 +0000 UTC m=+2022.980080997" watchObservedRunningTime="2026-10-30 20:03:43.089966477 +0000 UTC m=+2022.982580813"
Oct 30 20:03:43.445077 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:43.445035    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:43.453692 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:43.453667    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:44.072563 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:44.072524    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d" event={"ID":"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15","Type":"ContainerStarted","Data":"7888a3364ad8d80ae22a582ee90f584d2243846afbf27550d4277f74b76ed584"}
Oct 30 20:03:44.072743 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:44.072591    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d" event={"ID":"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15","Type":"ContainerStarted","Data":"b3bc9b939aec6cf61d331f64c325247570a012ba346f3e6b30e25850f5bb2ebc"}
Oct 30 20:03:44.073094 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:44.073036    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:44.074211 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:44.074191    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:03:44.106727 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:44.106684    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d" podStartSLOduration=3.106670848 podStartE2EDuration="3.106670848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:44.101135056 +0000 UTC m=+2023.993749380" watchObservedRunningTime="2026-10-30 20:03:44.106670848 +0000 UTC m=+2023.999285171"
Oct 30 20:03:44.340301 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:44.340205    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:44.340657 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:44.340638    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:45.081116 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:45.081077    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:45.082012 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:45.081991    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:03:47.840598 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:47.840560    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"]
Oct 30 20:03:47.845850 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:47.845802    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-96ec93-rxg4x"]
Oct 30 20:03:48.318097 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:48.318057    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28617b93-ee6f-4ec6-a3f1-e38018ea27e8" path="/var/lib/kubelet/pods/28617b93-ee6f-4ec6-a3f1-e38018ea27e8/volumes"
Oct 30 20:03:55.986830 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:55.986797    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"]
Oct 30 20:03:55.987360 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:03:55.987095    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9" podUID="35ad66e7-5505-4c0b-96f8-833adf2bd769" containerName="istio-proxy" containerID="cri-o://522c0e6ae0d7d02cafbaaba4eccb9e8c43ff52315c37ae185e22eb4833b4fd9d" gracePeriod=30
Oct 30 20:04:01.313955 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.313919    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:04:01.394143 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.394111    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-podinfo\") pod \"35ad66e7-5505-4c0b-96f8-833adf2bd769\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") "
Oct 30 20:04:01.394347 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.394257    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-data\") pod \"35ad66e7-5505-4c0b-96f8-833adf2bd769\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") "
Oct 30 20:04:01.394347 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.394302    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-workload-certs\") pod \"35ad66e7-5505-4c0b-96f8-833adf2bd769\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") "
Oct 30 20:04:01.394493 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.394441    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35ad66e7-5505-4c0b-96f8-833adf2bd769-kube-api-access-gqdxf\" (UniqueName: \"kubernetes.io/projected/35ad66e7-5505-4c0b-96f8-833adf2bd769-kube-api-access-gqdxf\") pod \"35ad66e7-5505-4c0b-96f8-833adf2bd769\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") "
Oct 30 20:04:01.394555 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.394519    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-envoy\") pod \"35ad66e7-5505-4c0b-96f8-833adf2bd769\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") "
Oct 30 20:04:01.394610 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.394541    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-data" pod "35ad66e7-5505-4c0b-96f8-833adf2bd769" (UID: "35ad66e7-5505-4c0b-96f8-833adf2bd769"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:01.394665 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.394621    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-token\" (UniqueName: \"kubernetes.io/projected/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-token\") pod \"35ad66e7-5505-4c0b-96f8-833adf2bd769\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") "
Oct 30 20:04:01.394665 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.394619    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-workload-certs" pod "35ad66e7-5505-4c0b-96f8-833adf2bd769" (UID: "35ad66e7-5505-4c0b-96f8-833adf2bd769"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:01.394776 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.394759    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35ad66e7-5505-4c0b-96f8-833adf2bd769-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ad66e7-5505-4c0b-96f8-833adf2bd769-istiod-ca-cert\") pod \"35ad66e7-5505-4c0b-96f8-833adf2bd769\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") "
Oct 30 20:04:01.394836 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.394795    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-credential-socket\") pod \"35ad66e7-5505-4c0b-96f8-833adf2bd769\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") "
Oct 30 20:04:01.394952 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.394907    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-workload-socket\") pod \"35ad66e7-5505-4c0b-96f8-833adf2bd769\" (UID: \"35ad66e7-5505-4c0b-96f8-833adf2bd769\") "
Oct 30 20:04:01.395166 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.395138    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35ad66e7-5505-4c0b-96f8-833adf2bd769-istiod-ca-cert" pod "35ad66e7-5505-4c0b-96f8-833adf2bd769" (UID: "35ad66e7-5505-4c0b-96f8-833adf2bd769"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:01.395429 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.395405    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-credential-socket" pod "35ad66e7-5505-4c0b-96f8-833adf2bd769" (UID: "35ad66e7-5505-4c0b-96f8-833adf2bd769"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:01.396082 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.396052    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:01.396183 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.396085    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:01.396183 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.396104    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ad66e7-5505-4c0b-96f8-833adf2bd769-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:01.396183 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.396121    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:01.396359 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.396207    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-workload-socket" pod "35ad66e7-5505-4c0b-96f8-833adf2bd769" (UID: "35ad66e7-5505-4c0b-96f8-833adf2bd769"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:01.396951 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.396923    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-podinfo" pod "35ad66e7-5505-4c0b-96f8-833adf2bd769" (UID: "35ad66e7-5505-4c0b-96f8-833adf2bd769"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:04:01.397211 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.397192    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ad66e7-5505-4c0b-96f8-833adf2bd769-kube-api-access-gqdxf" pod "35ad66e7-5505-4c0b-96f8-833adf2bd769" (UID: "35ad66e7-5505-4c0b-96f8-833adf2bd769"). InnerVolumeSpecName "kube-api-access-gqdxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:01.397282 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.397220    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-token" pod "35ad66e7-5505-4c0b-96f8-833adf2bd769" (UID: "35ad66e7-5505-4c0b-96f8-833adf2bd769"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:01.397531 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.397515    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-envoy" pod "35ad66e7-5505-4c0b-96f8-833adf2bd769" (UID: "35ad66e7-5505-4c0b-96f8-833adf2bd769"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:01.497489 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.497456    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:01.497489 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.497486    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:01.497696 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.497512    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ad66e7-5505-4c0b-96f8-833adf2bd769-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:01.497696 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.497522    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ad66e7-5505-4c0b-96f8-833adf2bd769-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:01.497696 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:01.497534    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqdxf\" (UniqueName: \"kubernetes.io/projected/35ad66e7-5505-4c0b-96f8-833adf2bd769-kube-api-access-gqdxf\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:02.005298 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:02.005243    4823 generic.go:356] "Generic (PLEG): container finished" podID="35ad66e7-5505-4c0b-96f8-833adf2bd769" containerID="522c0e6ae0d7d02cafbaaba4eccb9e8c43ff52315c37ae185e22eb4833b4fd9d" exitCode=0
Oct 30 20:04:02.005528 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:02.005407    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9" event={"ID":"35ad66e7-5505-4c0b-96f8-833adf2bd769","Type":"ContainerDied","Data":"522c0e6ae0d7d02cafbaaba4eccb9e8c43ff52315c37ae185e22eb4833b4fd9d"}
Oct 30 20:04:02.005528 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:02.005511    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9" event={"ID":"35ad66e7-5505-4c0b-96f8-833adf2bd769","Type":"ContainerDied","Data":"0dab86b4b1b3f2efd99a485e51eb0998c594a56abc26367b2cb04781701d6efb"}
Oct 30 20:04:02.005648 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:02.005552    4823 scope.go:122] "RemoveContainer" containerID="522c0e6ae0d7d02cafbaaba4eccb9e8c43ff52315c37ae185e22eb4833b4fd9d"
Oct 30 20:04:02.005809 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:02.005790    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"
Oct 30 20:04:02.038548 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:02.038530    4823 scope.go:122] "RemoveContainer" containerID="dccc357281ebafd063b095e06fb81e77a6fc66b00a9cd7a558b5ab49f7f09eb9"
Oct 30 20:04:02.058573 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:02.058537    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"]
Oct 30 20:04:02.058933 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:04:02.058907    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:04:02.059248 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:02.059231    4823 scope.go:122] "RemoveContainer" containerID="522c0e6ae0d7d02cafbaaba4eccb9e8c43ff52315c37ae185e22eb4833b4fd9d"
Oct 30 20:04:02.059604 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:04:02.059570    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522c0e6ae0d7d02cafbaaba4eccb9e8c43ff52315c37ae185e22eb4833b4fd9d\": container with ID starting with 522c0e6ae0d7d02cafbaaba4eccb9e8c43ff52315c37ae185e22eb4833b4fd9d not found: ID does not exist" containerID="522c0e6ae0d7d02cafbaaba4eccb9e8c43ff52315c37ae185e22eb4833b4fd9d"
Oct 30 20:04:02.059679 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:02.059615    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522c0e6ae0d7d02cafbaaba4eccb9e8c43ff52315c37ae185e22eb4833b4fd9d"} err="failed to get container status \"522c0e6ae0d7d02cafbaaba4eccb9e8c43ff52315c37ae185e22eb4833b4fd9d\": rpc error: code = NotFound desc = could not find container \"522c0e6ae0d7d02cafbaaba4eccb9e8c43ff52315c37ae185e22eb4833b4fd9d\": container with ID starting with 522c0e6ae0d7d02cafbaaba4eccb9e8c43ff52315c37ae185e22eb4833b4fd9d not found: ID does not exist"
Oct 30 20:04:02.059679 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:02.059640    4823 scope.go:122] "RemoveContainer" containerID="dccc357281ebafd063b095e06fb81e77a6fc66b00a9cd7a558b5ab49f7f09eb9"
Oct 30 20:04:02.060067 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:04:02.060039    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dccc357281ebafd063b095e06fb81e77a6fc66b00a9cd7a558b5ab49f7f09eb9\": container with ID starting with dccc357281ebafd063b095e06fb81e77a6fc66b00a9cd7a558b5ab49f7f09eb9 not found: ID does not exist" containerID="dccc357281ebafd063b095e06fb81e77a6fc66b00a9cd7a558b5ab49f7f09eb9"
Oct 30 20:04:02.060137 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:02.060068    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dccc357281ebafd063b095e06fb81e77a6fc66b00a9cd7a558b5ab49f7f09eb9"} err="failed to get container status \"dccc357281ebafd063b095e06fb81e77a6fc66b00a9cd7a558b5ab49f7f09eb9\": rpc error: code = NotFound desc = could not find container \"dccc357281ebafd063b095e06fb81e77a6fc66b00a9cd7a558b5ab49f7f09eb9\": container with ID starting with dccc357281ebafd063b095e06fb81e77a6fc66b00a9cd7a558b5ab49f7f09eb9 not found: ID does not exist"
Oct 30 20:04:02.064500 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:02.064475    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-96ec93-openshift-default-74695cc7-qdfj9"]
Oct 30 20:04:02.105945 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:02.105824    4823 scope.go:122] "RemoveContainer" containerID="01e305f5e77f7fef928d0a05d8ff0803bd8dbe8cc4f537d841054ef22c13c93a"
Oct 30 20:04:02.281017 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:02.280939    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35ad66e7-5505-4c0b-96f8-833adf2bd769" path="/var/lib/kubelet/pods/35ad66e7-5505-4c0b-96f8-833adf2bd769/volumes"
Oct 30 20:04:06.551886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:06.551721    4823 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:04:06.579984 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:06.579943    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-9djkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:04:06.580284 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:06.580251    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx" podUID="88deb5dd-bf6b-406b-9aa9-103319a550e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:04:06.836941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:06.835993    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:04:06.837136 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:06.837083    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 20:04:07.076886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:07.075969    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:04:07.221017 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:07.220967    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:04:09.934889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:09.930913    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"]
Oct 30 20:04:09.934889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:09.932261    4823 cpu_manager.go:414] "containerMap: removing container" podUID="35ad66e7-5505-4c0b-96f8-833adf2bd769" containerName="istio-proxy"
Oct 30 20:04:09.934889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:09.932281    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35ad66e7-5505-4c0b-96f8-833adf2bd769" containerName="istio-proxy"
Oct 30 20:04:09.934889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:09.932512    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="35ad66e7-5505-4c0b-96f8-833adf2bd769" containerName="istio-proxy"
Oct 30 20:04:09.968243 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:09.968204    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:04:09.986910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:09.973937    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"]
Oct 30 20:04:09.986910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:09.974178    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d7a733-serving-cert\""
Oct 30 20:04:10.070448 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:10.070392    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb403b1-bc1d-4f49-88e1-fbb1baceae02-metrics-certs\") pod \"maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb\" (UID: \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:04:10.070907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:10.070849    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb403b1-bc1d-4f49-88e1-fbb1baceae02-maas-api-tls\") pod \"maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb\" (UID: \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:04:10.071285 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:10.071249    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftzk7\" (UniqueName: \"kubernetes.io/projected/adb403b1-bc1d-4f49-88e1-fbb1baceae02-kube-api-access-ftzk7\") pod \"maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb\" (UID: \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:04:10.071563 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:10.071536    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb403b1-bc1d-4f49-88e1-fbb1baceae02-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb\" (UID: \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:04:10.198936 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:10.198665    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb403b1-bc1d-4f49-88e1-fbb1baceae02-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb\" (UID: \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:04:10.198936 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:10.198849    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb403b1-bc1d-4f49-88e1-fbb1baceae02-metrics-certs\") pod \"maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb\" (UID: \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:04:10.199207 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:10.199125    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb403b1-bc1d-4f49-88e1-fbb1baceae02-maas-api-tls\") pod \"maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb\" (UID: \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:04:10.245882 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:10.199380    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftzk7\" (UniqueName: \"kubernetes.io/projected/adb403b1-bc1d-4f49-88e1-fbb1baceae02-kube-api-access-ftzk7\") pod \"maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb\" (UID: \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:04:10.245882 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:10.216977    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb403b1-bc1d-4f49-88e1-fbb1baceae02-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb\" (UID: \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:04:10.256688 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:10.249193    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb403b1-bc1d-4f49-88e1-fbb1baceae02-maas-api-tls\") pod \"maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb\" (UID: \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:04:10.292845 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:10.292467    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb403b1-bc1d-4f49-88e1-fbb1baceae02-metrics-certs\") pod \"maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb\" (UID: \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:04:10.304148 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:10.304087    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftzk7\" (UniqueName: \"kubernetes.io/projected/adb403b1-bc1d-4f49-88e1-fbb1baceae02-kube-api-access-ftzk7\") pod \"maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb\" (UID: \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:04:10.351508 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:10.351480    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:04:10.650115 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:10.650090    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"]
Oct 30 20:04:10.653001 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:04:10.652966    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb403b1_bc1d_4f49_88e1_fbb1baceae02.slice/crio-526aca620a9ffa6d7d009db2f351be29fdd003bec38061f0e7cedefc314b0459 WatchSource:0}: Error finding container 526aca620a9ffa6d7d009db2f351be29fdd003bec38061f0e7cedefc314b0459: Status 404 returned error can't find the container with id 526aca620a9ffa6d7d009db2f351be29fdd003bec38061f0e7cedefc314b0459
Oct 30 20:04:11.323842 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:11.323799    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb" event={"ID":"adb403b1-bc1d-4f49-88e1-fbb1baceae02","Type":"ContainerStarted","Data":"711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3"}
Oct 30 20:04:11.324360 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:11.323940    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb" event={"ID":"adb403b1-bc1d-4f49-88e1-fbb1baceae02","Type":"ContainerStarted","Data":"526aca620a9ffa6d7d009db2f351be29fdd003bec38061f0e7cedefc314b0459"}
Oct 30 20:04:11.324360 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:11.324184    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:04:11.346300 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:11.346244    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb" podStartSLOduration=1.797746113 podStartE2EDuration="2.346225686s" totalImagesPullingTime="548.479573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:11.34382874 +0000 UTC m=+2051.236443066" watchObservedRunningTime="2026-10-30 20:04:11.346225686 +0000 UTC m=+2051.238840011"
Oct 30 20:04:14.945941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:14.942196    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"]
Oct 30 20:04:14.945941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:14.943121    4823 cpu_manager.go:414] "containerMap: removing container" podUID="35ad66e7-5505-4c0b-96f8-833adf2bd769" containerName="istio-proxy"
Oct 30 20:04:14.945941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:14.943137    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35ad66e7-5505-4c0b-96f8-833adf2bd769" containerName="istio-proxy"
Oct 30 20:04:14.945941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:14.943259    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="35ad66e7-5505-4c0b-96f8-833adf2bd769" containerName="istio-proxy"
Oct 30 20:04:14.947317 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:14.947292    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:04:14.958940 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:14.956978    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-925e04-serving-cert\""
Oct 30 20:04:14.991195 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:14.991162    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"]
Oct 30 20:04:15.104474 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:15.104433    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-maas-api-tls\") pod \"maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz\" (UID: \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:04:15.104657 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:15.104502    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-metrics-certs\") pod \"maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz\" (UID: \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:04:15.104762 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:15.104738    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz\" (UID: \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:04:15.105023 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:15.105000    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnlxh\" (UniqueName: \"kubernetes.io/projected/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-kube-api-access-qnlxh\") pod \"maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz\" (UID: \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:04:15.209983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:15.207620    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz\" (UID: \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:04:15.209983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:15.207912    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnlxh\" (UniqueName: \"kubernetes.io/projected/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-kube-api-access-qnlxh\") pod \"maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz\" (UID: \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:04:15.209983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:15.207950    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-maas-api-tls\") pod \"maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz\" (UID: \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:04:15.209983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:15.208006    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-metrics-certs\") pod \"maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz\" (UID: \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:04:15.218986 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:15.218950    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz\" (UID: \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:04:15.257655 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:15.257616    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnlxh\" (UniqueName: \"kubernetes.io/projected/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-kube-api-access-qnlxh\") pod \"maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz\" (UID: \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:04:15.285533 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:15.285488    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-maas-api-tls\") pod \"maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz\" (UID: \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:04:15.286276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:15.286209    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-metrics-certs\") pod \"maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz\" (UID: \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:04:15.568514 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:15.568434    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:04:16.092594 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:04:16.092560    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efe25a4_1f89_4f9d_b6f2_bc14c344e69b.slice/crio-536c8547cb5bafe1e95b0f9f00824cf88bb62d647052bccb604e1b75ceb97240 WatchSource:0}: Error finding container 536c8547cb5bafe1e95b0f9f00824cf88bb62d647052bccb604e1b75ceb97240: Status 404 returned error can't find the container with id 536c8547cb5bafe1e95b0f9f00824cf88bb62d647052bccb604e1b75ceb97240
Oct 30 20:04:16.093890 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:16.093847    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"]
Oct 30 20:04:16.369576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:16.369541    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz" event={"ID":"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b","Type":"ContainerStarted","Data":"536c8547cb5bafe1e95b0f9f00824cf88bb62d647052bccb604e1b75ceb97240"}
Oct 30 20:04:17.339592 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:17.339562    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:04:17.380768 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:17.380734    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz" event={"ID":"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b","Type":"ContainerStarted","Data":"69a9f99f527b0514999ce6d899dfc2fdf74d58ce6f3f80ac2814407578955dfc"}
Oct 30 20:04:17.381622 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:17.381601    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:04:17.455794 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:17.455725    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz" podStartSLOduration=2.942000685 podStartE2EDuration="3.455704322s" totalImagesPullingTime="513.703637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:17.45407074 +0000 UTC m=+2057.346685065" watchObservedRunningTime="2026-10-30 20:04:17.455704322 +0000 UTC m=+2057.348318647"
Oct 30 20:04:18.049962 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:18.049932    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"]
Oct 30 20:04:18.054508 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:18.054486    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2cee2b0-2zfw4"]
Oct 30 20:04:18.341944 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:18.339001    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b10a3f-398f-4228-be67-f92292d43b6f" path="/var/lib/kubelet/pods/83b10a3f-398f-4228-be67-f92292d43b6f/volumes"
Oct 30 20:04:20.487376 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:20.483220    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"]
Oct 30 20:04:20.488259 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:20.488185    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw" podUID="4379a4ca-7279-4f32-9b90-39baca931c17" containerName="istio-proxy" containerID="cri-o://eebe8c7f934783d682b69ecdb302391fef6f3d166baf46789aae1912bef4c26d" gracePeriod=30
Oct 30 20:04:24.441607 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:24.441572    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:04:25.730695 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:25.730660    4823 generic.go:356] "Generic (PLEG): container finished" podID="4379a4ca-7279-4f32-9b90-39baca931c17" containerID="eebe8c7f934783d682b69ecdb302391fef6f3d166baf46789aae1912bef4c26d" exitCode=0
Oct 30 20:04:25.731138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:25.730994    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw" event={"ID":"4379a4ca-7279-4f32-9b90-39baca931c17","Type":"ContainerDied","Data":"eebe8c7f934783d682b69ecdb302391fef6f3d166baf46789aae1912bef4c26d"}
Oct 30 20:04:25.864925 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:25.864897    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 20:04:26.018567 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.018529    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4379a4ca-7279-4f32-9b90-39baca931c17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4379a4ca-7279-4f32-9b90-39baca931c17-istio-podinfo\") pod \"4379a4ca-7279-4f32-9b90-39baca931c17\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") "
Oct 30 20:04:26.018786 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.018587    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4379a4ca-7279-4f32-9b90-39baca931c17-istio-token\" (UniqueName: \"kubernetes.io/projected/4379a4ca-7279-4f32-9b90-39baca931c17-istio-token\") pod \"4379a4ca-7279-4f32-9b90-39baca931c17\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") "
Oct 30 20:04:26.018786 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.018690    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-credential-socket\") pod \"4379a4ca-7279-4f32-9b90-39baca931c17\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") "
Oct 30 20:04:26.018967 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.018792    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4379a4ca-7279-4f32-9b90-39baca931c17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4379a4ca-7279-4f32-9b90-39baca931c17-istiod-ca-cert\") pod \"4379a4ca-7279-4f32-9b90-39baca931c17\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") "
Oct 30 20:04:26.019040 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.019011    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4379a4ca-7279-4f32-9b90-39baca931c17-kube-api-access-l7gdh\" (UniqueName: \"kubernetes.io/projected/4379a4ca-7279-4f32-9b90-39baca931c17-kube-api-access-l7gdh\") pod \"4379a4ca-7279-4f32-9b90-39baca931c17\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") "
Oct 30 20:04:26.019093 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.019043    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-workload-certs\") pod \"4379a4ca-7279-4f32-9b90-39baca931c17\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") "
Oct 30 20:04:26.019156 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.019102    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-credential-socket" pod "4379a4ca-7279-4f32-9b90-39baca931c17" (UID: "4379a4ca-7279-4f32-9b90-39baca931c17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:26.019207 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.019156    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-istio-data\") pod \"4379a4ca-7279-4f32-9b90-39baca931c17\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") "
Oct 30 20:04:26.019262 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.019232    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-istio-envoy\") pod \"4379a4ca-7279-4f32-9b90-39baca931c17\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") "
Oct 30 20:04:26.019313 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.019299    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-workload-socket\") pod \"4379a4ca-7279-4f32-9b90-39baca931c17\" (UID: \"4379a4ca-7279-4f32-9b90-39baca931c17\") "
Oct 30 20:04:26.019449 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.019425    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4379a4ca-7279-4f32-9b90-39baca931c17-istiod-ca-cert" pod "4379a4ca-7279-4f32-9b90-39baca931c17" (UID: "4379a4ca-7279-4f32-9b90-39baca931c17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:26.019687 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.019631    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-istio-data" pod "4379a4ca-7279-4f32-9b90-39baca931c17" (UID: "4379a4ca-7279-4f32-9b90-39baca931c17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:26.019873 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.019829    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-workload-certs" pod "4379a4ca-7279-4f32-9b90-39baca931c17" (UID: "4379a4ca-7279-4f32-9b90-39baca931c17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:26.020363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.019960    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-workload-socket" pod "4379a4ca-7279-4f32-9b90-39baca931c17" (UID: "4379a4ca-7279-4f32-9b90-39baca931c17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:26.020363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.020261    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4379a4ca-7279-4f32-9b90-39baca931c17-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:26.020363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.020276    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:26.020363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.020285    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:26.020363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.020297    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:26.020363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.020312    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:26.021453 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.021426    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4379a4ca-7279-4f32-9b90-39baca931c17-istio-podinfo" pod "4379a4ca-7279-4f32-9b90-39baca931c17" (UID: "4379a4ca-7279-4f32-9b90-39baca931c17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:04:26.022103 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.022073    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-istio-envoy" pod "4379a4ca-7279-4f32-9b90-39baca931c17" (UID: "4379a4ca-7279-4f32-9b90-39baca931c17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:26.022208 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.022097    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4379a4ca-7279-4f32-9b90-39baca931c17-kube-api-access-l7gdh" pod "4379a4ca-7279-4f32-9b90-39baca931c17" (UID: "4379a4ca-7279-4f32-9b90-39baca931c17"). InnerVolumeSpecName "kube-api-access-l7gdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:26.022208 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.022114    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4379a4ca-7279-4f32-9b90-39baca931c17-istio-token" pod "4379a4ca-7279-4f32-9b90-39baca931c17" (UID: "4379a4ca-7279-4f32-9b90-39baca931c17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:26.121763 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.121727    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7gdh\" (UniqueName: \"kubernetes.io/projected/4379a4ca-7279-4f32-9b90-39baca931c17-kube-api-access-l7gdh\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:26.121935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.121769    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4379a4ca-7279-4f32-9b90-39baca931c17-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:26.121935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.121785    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4379a4ca-7279-4f32-9b90-39baca931c17-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:26.121935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.121797    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4379a4ca-7279-4f32-9b90-39baca931c17-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:26.742154 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.742125    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"
Oct 30 20:04:26.742639 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.742146    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw" event={"ID":"4379a4ca-7279-4f32-9b90-39baca931c17","Type":"ContainerDied","Data":"108622826baacabfadcc48c16514cd4baa5156afe360b5b320cfa45cd0847dc5"}
Oct 30 20:04:26.742639 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.742227    4823 scope.go:122] "RemoveContainer" containerID="eebe8c7f934783d682b69ecdb302391fef6f3d166baf46789aae1912bef4c26d"
Oct 30 20:04:26.763679 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.763641    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"]
Oct 30 20:04:26.768214 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:26.768190    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-6pjrw"]
Oct 30 20:04:27.375621 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:27.375587    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"]
Oct 30 20:04:27.376134 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:27.376099    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z" podUID="118587b4-0b24-4a51-b462-68bce9602a0a" containerName="istio-proxy" containerID="cri-o://4d2f5bb04ebd505550e9415566f5fca3e4525d08207da928065ccb63134a9bad" gracePeriod=30
Oct 30 20:04:27.530423 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:27.530386    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"]
Oct 30 20:04:27.530661 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:27.530632    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" containerID="cri-o://05f06ff70606adbb5a5c1bd86284a755dade95ec61dee2c35e6a144fae263597" gracePeriod=30
Oct 30 20:04:28.286610 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:28.286575    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4379a4ca-7279-4f32-9b90-39baca931c17" path="/var/lib/kubelet/pods/4379a4ca-7279-4f32-9b90-39baca931c17/volumes"
Oct 30 20:04:28.573266 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:28.573178    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:04:28.573446 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:28.573274    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:04:31.404599 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.404563    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"]
Oct 30 20:04:31.405830 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.405810    4823 cpu_manager.go:414] "containerMap: removing container" podUID="4379a4ca-7279-4f32-9b90-39baca931c17" containerName="istio-proxy"
Oct 30 20:04:31.405959 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.405832    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4379a4ca-7279-4f32-9b90-39baca931c17" containerName="istio-proxy"
Oct 30 20:04:31.406060 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.406048    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="4379a4ca-7279-4f32-9b90-39baca931c17" containerName="istio-proxy"
Oct 30 20:04:31.413559 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.413535    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:04:31.428887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.428671    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7f7d21-serving-cert\""
Oct 30 20:04:31.433909 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.433597    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"]
Oct 30 20:04:31.471613 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.471576    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f67270-49ff-4e06-988f-84deb2909f8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq\" (UID: \"20f67270-49ff-4e06-988f-84deb2909f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:04:31.471986 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.471820    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnzt5\" (UniqueName: \"kubernetes.io/projected/20f67270-49ff-4e06-988f-84deb2909f8e-kube-api-access-jnzt5\") pod \"maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq\" (UID: \"20f67270-49ff-4e06-988f-84deb2909f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:04:31.472110 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.472014    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f67270-49ff-4e06-988f-84deb2909f8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq\" (UID: \"20f67270-49ff-4e06-988f-84deb2909f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:04:31.472110 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.472043    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f67270-49ff-4e06-988f-84deb2909f8e-metrics-certs\") pod \"maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq\" (UID: \"20f67270-49ff-4e06-988f-84deb2909f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:04:31.573725 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.573692    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnzt5\" (UniqueName: \"kubernetes.io/projected/20f67270-49ff-4e06-988f-84deb2909f8e-kube-api-access-jnzt5\") pod \"maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq\" (UID: \"20f67270-49ff-4e06-988f-84deb2909f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:04:31.573979 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.573789    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f67270-49ff-4e06-988f-84deb2909f8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq\" (UID: \"20f67270-49ff-4e06-988f-84deb2909f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:04:31.573979 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.573819    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f67270-49ff-4e06-988f-84deb2909f8e-metrics-certs\") pod \"maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq\" (UID: \"20f67270-49ff-4e06-988f-84deb2909f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:04:31.574124 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.574106    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f67270-49ff-4e06-988f-84deb2909f8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq\" (UID: \"20f67270-49ff-4e06-988f-84deb2909f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:04:31.574597 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.574572    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f67270-49ff-4e06-988f-84deb2909f8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq\" (UID: \"20f67270-49ff-4e06-988f-84deb2909f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:04:31.576613 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.576592    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f67270-49ff-4e06-988f-84deb2909f8e-metrics-certs\") pod \"maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq\" (UID: \"20f67270-49ff-4e06-988f-84deb2909f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:04:31.576688 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.576620    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f67270-49ff-4e06-988f-84deb2909f8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq\" (UID: \"20f67270-49ff-4e06-988f-84deb2909f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:04:31.586935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.586911    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnzt5\" (UniqueName: \"kubernetes.io/projected/20f67270-49ff-4e06-988f-84deb2909f8e-kube-api-access-jnzt5\") pod \"maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq\" (UID: \"20f67270-49ff-4e06-988f-84deb2909f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:04:31.746635 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:31.746549    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:04:32.127904 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.124552    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"]
Oct 30 20:04:32.201163 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:04:32.201122    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f67270_49ff_4e06_988f_84deb2909f8e.slice/crio-8f6f2db01433ddfe840c1570906e2d0d03113d6286af2ae682dec00bafac2cab WatchSource:0}: Error finding container 8f6f2db01433ddfe840c1570906e2d0d03113d6286af2ae682dec00bafac2cab: Status 404 returned error can't find the container with id 8f6f2db01433ddfe840c1570906e2d0d03113d6286af2ae682dec00bafac2cab
Oct 30 20:04:32.219086 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:04:32.219048    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:04:32.796497 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.796069    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 20:04:32.818051 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.817751    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq" event={"ID":"20f67270-49ff-4e06-988f-84deb2909f8e","Type":"ContainerStarted","Data":"8f6f2db01433ddfe840c1570906e2d0d03113d6286af2ae682dec00bafac2cab"}
Oct 30 20:04:32.832910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.832232    4823 generic.go:356] "Generic (PLEG): container finished" podID="118587b4-0b24-4a51-b462-68bce9602a0a" containerID="4d2f5bb04ebd505550e9415566f5fca3e4525d08207da928065ccb63134a9bad" exitCode=0
Oct 30 20:04:32.832910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.832413    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z" event={"ID":"118587b4-0b24-4a51-b462-68bce9602a0a","Type":"ContainerDied","Data":"4d2f5bb04ebd505550e9415566f5fca3e4525d08207da928065ccb63134a9bad"}
Oct 30 20:04:32.832910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.832465    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z" event={"ID":"118587b4-0b24-4a51-b462-68bce9602a0a","Type":"ContainerDied","Data":"4f89c7df5d895b4bae76089d758980690f2d639a33b8918d0d3f19ff6dd28d68"}
Oct 30 20:04:32.832910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.832495    4823 scope.go:122] "RemoveContainer" containerID="4d2f5bb04ebd505550e9415566f5fca3e4525d08207da928065ccb63134a9bad"
Oct 30 20:04:32.832910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.832649    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"
Oct 30 20:04:32.840704 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.840678    4823 generic.go:356] "Generic (PLEG): container finished" podID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerID="05f06ff70606adbb5a5c1bd86284a755dade95ec61dee2c35e6a144fae263597" exitCode=0
Oct 30 20:04:32.840958 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.840787    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" event={"ID":"ee4bdb13-b5bb-4692-bbff-769a263dcffa","Type":"ContainerDied","Data":"05f06ff70606adbb5a5c1bd86284a755dade95ec61dee2c35e6a144fae263597"}
Oct 30 20:04:32.851081 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.851057    4823 scope.go:122] "RemoveContainer" containerID="4d2f5bb04ebd505550e9415566f5fca3e4525d08207da928065ccb63134a9bad"
Oct 30 20:04:32.851968 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:04:32.851935    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d2f5bb04ebd505550e9415566f5fca3e4525d08207da928065ccb63134a9bad\": container with ID starting with 4d2f5bb04ebd505550e9415566f5fca3e4525d08207da928065ccb63134a9bad not found: ID does not exist" containerID="4d2f5bb04ebd505550e9415566f5fca3e4525d08207da928065ccb63134a9bad"
Oct 30 20:04:32.852063 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.851978    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d2f5bb04ebd505550e9415566f5fca3e4525d08207da928065ccb63134a9bad"} err="failed to get container status \"4d2f5bb04ebd505550e9415566f5fca3e4525d08207da928065ccb63134a9bad\": rpc error: code = NotFound desc = could not find container \"4d2f5bb04ebd505550e9415566f5fca3e4525d08207da928065ccb63134a9bad\": container with ID starting with 4d2f5bb04ebd505550e9415566f5fca3e4525d08207da928065ccb63134a9bad not found: ID does not exist"
Oct 30 20:04:32.895057 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.895006    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/118587b4-0b24-4a51-b462-68bce9602a0a-istio-token\" (UniqueName: \"kubernetes.io/projected/118587b4-0b24-4a51-b462-68bce9602a0a-istio-token\") pod \"118587b4-0b24-4a51-b462-68bce9602a0a\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") "
Oct 30 20:04:32.895769 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.895748    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-workload-certs\") pod \"118587b4-0b24-4a51-b462-68bce9602a0a\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") "
Oct 30 20:04:32.896168 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.896130    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-workload-certs" pod "118587b4-0b24-4a51-b462-68bce9602a0a" (UID: "118587b4-0b24-4a51-b462-68bce9602a0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:32.896305 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.896161    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-istio-data\") pod \"118587b4-0b24-4a51-b462-68bce9602a0a\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") "
Oct 30 20:04:32.896650 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.896627    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/118587b4-0b24-4a51-b462-68bce9602a0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/118587b4-0b24-4a51-b462-68bce9602a0a-istio-podinfo\") pod \"118587b4-0b24-4a51-b462-68bce9602a0a\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") "
Oct 30 20:04:32.896800 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.896784    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-credential-socket\") pod \"118587b4-0b24-4a51-b462-68bce9602a0a\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") "
Oct 30 20:04:32.896965 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.896948    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-workload-socket\") pod \"118587b4-0b24-4a51-b462-68bce9602a0a\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") "
Oct 30 20:04:32.897135 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.897115    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/118587b4-0b24-4a51-b462-68bce9602a0a-kube-api-access-jn26p\" (UniqueName: \"kubernetes.io/projected/118587b4-0b24-4a51-b462-68bce9602a0a-kube-api-access-jn26p\") pod \"118587b4-0b24-4a51-b462-68bce9602a0a\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") "
Oct 30 20:04:32.897340 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.897307    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/118587b4-0b24-4a51-b462-68bce9602a0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/118587b4-0b24-4a51-b462-68bce9602a0a-istiod-ca-cert\") pod \"118587b4-0b24-4a51-b462-68bce9602a0a\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") "
Oct 30 20:04:32.897509 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.897491    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-istio-envoy\") pod \"118587b4-0b24-4a51-b462-68bce9602a0a\" (UID: \"118587b4-0b24-4a51-b462-68bce9602a0a\") "
Oct 30 20:04:32.897919 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.896444    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-istio-data" pod "118587b4-0b24-4a51-b462-68bce9602a0a" (UID: "118587b4-0b24-4a51-b462-68bce9602a0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:32.897919 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.897432    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-credential-socket" pod "118587b4-0b24-4a51-b462-68bce9602a0a" (UID: "118587b4-0b24-4a51-b462-68bce9602a0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:32.897919 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.897587    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-workload-socket" pod "118587b4-0b24-4a51-b462-68bce9602a0a" (UID: "118587b4-0b24-4a51-b462-68bce9602a0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:32.903939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.898289    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118587b4-0b24-4a51-b462-68bce9602a0a-istiod-ca-cert" pod "118587b4-0b24-4a51-b462-68bce9602a0a" (UID: "118587b4-0b24-4a51-b462-68bce9602a0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:32.903939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.898651    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:32.903939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.898676    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:32.903939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.898692    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:32.903939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.898706    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:32.903939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.898720    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/118587b4-0b24-4a51-b462-68bce9602a0a-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:32.903939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.900822    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/118587b4-0b24-4a51-b462-68bce9602a0a-istio-podinfo" pod "118587b4-0b24-4a51-b462-68bce9602a0a" (UID: "118587b4-0b24-4a51-b462-68bce9602a0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:04:32.903939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.903050    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118587b4-0b24-4a51-b462-68bce9602a0a-istio-token" pod "118587b4-0b24-4a51-b462-68bce9602a0a" (UID: "118587b4-0b24-4a51-b462-68bce9602a0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:32.905144 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.905119    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118587b4-0b24-4a51-b462-68bce9602a0a-kube-api-access-jn26p" pod "118587b4-0b24-4a51-b462-68bce9602a0a" (UID: "118587b4-0b24-4a51-b462-68bce9602a0a"). InnerVolumeSpecName "kube-api-access-jn26p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:32.907219 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.907180    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-istio-envoy" pod "118587b4-0b24-4a51-b462-68bce9602a0a" (UID: "118587b4-0b24-4a51-b462-68bce9602a0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:32.973932 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:32.973854    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 20:04:33.000514 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.000479    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn26p\" (UniqueName: \"kubernetes.io/projected/118587b4-0b24-4a51-b462-68bce9602a0a-kube-api-access-jn26p\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.000753 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.000724    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/118587b4-0b24-4a51-b462-68bce9602a0a-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.000753 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.000753    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/118587b4-0b24-4a51-b462-68bce9602a0a-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.001024 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.000768    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/118587b4-0b24-4a51-b462-68bce9602a0a-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.102194 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.102155    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4bdb13-b5bb-4692-bbff-769a263dcffa-kube-api-access-jnnz7\" (UniqueName: \"kubernetes.io/projected/ee4bdb13-b5bb-4692-bbff-769a263dcffa-kube-api-access-jnnz7\") pod \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") "
Oct 30 20:04:33.102404 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.102316    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-podinfo\") pod \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") "
Oct 30 20:04:33.102404 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.102353    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-workload-certs\") pod \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") "
Oct 30 20:04:33.102523 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.102417    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istiod-ca-cert\") pod \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") "
Oct 30 20:04:33.102523 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.102465    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-envoy\") pod \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") "
Oct 30 20:04:33.102725 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.102697    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-workload-certs" pod "ee4bdb13-b5bb-4692-bbff-769a263dcffa" (UID: "ee4bdb13-b5bb-4692-bbff-769a263dcffa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:33.102802 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.102769    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-data\") pod \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") "
Oct 30 20:04:33.102875 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.102794    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istiod-ca-cert" pod "ee4bdb13-b5bb-4692-bbff-769a263dcffa" (UID: "ee4bdb13-b5bb-4692-bbff-769a263dcffa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:33.102875 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.102853    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-workload-socket\") pod \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") "
Oct 30 20:04:33.102989 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.102897    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-credential-socket\") pod \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") "
Oct 30 20:04:33.102989 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.102957    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-token\" (UniqueName: \"kubernetes.io/projected/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-token\") pod \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\" (UID: \"ee4bdb13-b5bb-4692-bbff-769a263dcffa\") "
Oct 30 20:04:33.103098 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.103028    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-data" pod "ee4bdb13-b5bb-4692-bbff-769a263dcffa" (UID: "ee4bdb13-b5bb-4692-bbff-769a263dcffa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:33.103098 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.103066    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-workload-socket" pod "ee4bdb13-b5bb-4692-bbff-769a263dcffa" (UID: "ee4bdb13-b5bb-4692-bbff-769a263dcffa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:33.103299 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.103276    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-credential-socket" pod "ee4bdb13-b5bb-4692-bbff-769a263dcffa" (UID: "ee4bdb13-b5bb-4692-bbff-769a263dcffa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:33.103808 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.103778    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.103961 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.103813    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.103961 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.103832    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.103961 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.103847    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.103961 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.103884    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.104671 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.104639    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4bdb13-b5bb-4692-bbff-769a263dcffa-kube-api-access-jnnz7" pod "ee4bdb13-b5bb-4692-bbff-769a263dcffa" (UID: "ee4bdb13-b5bb-4692-bbff-769a263dcffa"). InnerVolumeSpecName "kube-api-access-jnnz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:33.104982 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.104946    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-podinfo" pod "ee4bdb13-b5bb-4692-bbff-769a263dcffa" (UID: "ee4bdb13-b5bb-4692-bbff-769a263dcffa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:04:33.104982 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.104950    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-envoy" pod "ee4bdb13-b5bb-4692-bbff-769a263dcffa" (UID: "ee4bdb13-b5bb-4692-bbff-769a263dcffa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:33.105276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.105250    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-token" pod "ee4bdb13-b5bb-4692-bbff-769a263dcffa" (UID: "ee4bdb13-b5bb-4692-bbff-769a263dcffa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:33.183537 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.183502    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"]
Oct 30 20:04:33.191095 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.191062    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-wxx6z"]
Oct 30 20:04:33.206077 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.206044    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.206241 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.206080    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnnz7\" (UniqueName: \"kubernetes.io/projected/ee4bdb13-b5bb-4692-bbff-769a263dcffa-kube-api-access-jnnz7\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.206241 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.206098    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.206241 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.206113    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4bdb13-b5bb-4692-bbff-769a263dcffa-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:33.851291 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.851257    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq" event={"ID":"20f67270-49ff-4e06-988f-84deb2909f8e","Type":"ContainerStarted","Data":"21eb596155b1ad11f0795305861c3330667ba6deef35d542821a90816c38b0cd"}
Oct 30 20:04:33.851779 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.851342    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:04:33.853878 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.853846    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"
Oct 30 20:04:33.854002 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.853889    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2" event={"ID":"ee4bdb13-b5bb-4692-bbff-769a263dcffa","Type":"ContainerDied","Data":"b043c5be1c925c66107117c2b5d8d8a5ee152d3c4a98a9dc7d79d0682210349b"}
Oct 30 20:04:33.854002 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.853943    4823 scope.go:122] "RemoveContainer" containerID="05f06ff70606adbb5a5c1bd86284a755dade95ec61dee2c35e6a144fae263597"
Oct 30 20:04:33.889444 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.889384    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq" podStartSLOduration=2.291651722 podStartE2EDuration="2.889370237s" totalImagesPullingTime="597.718515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:33.883431163 +0000 UTC m=+2073.776045487" watchObservedRunningTime="2026-10-30 20:04:33.889370237 +0000 UTC m=+2073.781984561"
Oct 30 20:04:33.932985 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.932952    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"]
Oct 30 20:04:33.945957 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:33.945928    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-rlck2"]
Oct 30 20:04:34.367443 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:34.367396    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="118587b4-0b24-4a51-b462-68bce9602a0a" path="/var/lib/kubelet/pods/118587b4-0b24-4a51-b462-68bce9602a0a/volumes"
Oct 30 20:04:34.368508 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:34.368475    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" path="/var/lib/kubelet/pods/ee4bdb13-b5bb-4692-bbff-769a263dcffa/volumes"
Oct 30 20:04:35.428258 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:35.428221    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"]
Oct 30 20:04:35.428656 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:35.428440    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc" podUID="44eb96c1-778e-433f-ae45-3a42beed5dac" containerName="istio-proxy" containerID="cri-o://671e1f1c862980d941a1747b1d9975fcf00760e7b94e103e1af167f3e9a1ad96" gracePeriod=30
Oct 30 20:04:39.896897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:39.895983    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:04:40.744740 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.744711    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 20:04:40.846495 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.846465    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-envoy\") pod \"44eb96c1-778e-433f-ae45-3a42beed5dac\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") "
Oct 30 20:04:40.846495 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.846499    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-credential-socket\") pod \"44eb96c1-778e-433f-ae45-3a42beed5dac\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") "
Oct 30 20:04:40.846749 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.846537    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44eb96c1-778e-433f-ae45-3a42beed5dac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eb96c1-778e-433f-ae45-3a42beed5dac-istiod-ca-cert\") pod \"44eb96c1-778e-433f-ae45-3a42beed5dac\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") "
Oct 30 20:04:40.846749 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.846578    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-data\") pod \"44eb96c1-778e-433f-ae45-3a42beed5dac\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") "
Oct 30 20:04:40.846749 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.846603    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-podinfo\") pod \"44eb96c1-778e-433f-ae45-3a42beed5dac\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") "
Oct 30 20:04:40.846749 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.846693    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-workload-certs\") pod \"44eb96c1-778e-433f-ae45-3a42beed5dac\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") "
Oct 30 20:04:40.846749 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.846741    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44eb96c1-778e-433f-ae45-3a42beed5dac-kube-api-access-w4fgh\" (UniqueName: \"kubernetes.io/projected/44eb96c1-778e-433f-ae45-3a42beed5dac-kube-api-access-w4fgh\") pod \"44eb96c1-778e-433f-ae45-3a42beed5dac\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") "
Oct 30 20:04:40.847057 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.846819    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-credential-socket" pod "44eb96c1-778e-433f-ae45-3a42beed5dac" (UID: "44eb96c1-778e-433f-ae45-3a42beed5dac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:40.847419 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.847390    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-workload-certs" pod "44eb96c1-778e-433f-ae45-3a42beed5dac" (UID: "44eb96c1-778e-433f-ae45-3a42beed5dac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:40.847540 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.847422    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-token\" (UniqueName: \"kubernetes.io/projected/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-token\") pod \"44eb96c1-778e-433f-ae45-3a42beed5dac\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") "
Oct 30 20:04:40.847540 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.847459    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-data" pod "44eb96c1-778e-433f-ae45-3a42beed5dac" (UID: "44eb96c1-778e-433f-ae45-3a42beed5dac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:40.847540 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.847482    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-workload-socket\") pod \"44eb96c1-778e-433f-ae45-3a42beed5dac\" (UID: \"44eb96c1-778e-433f-ae45-3a42beed5dac\") "
Oct 30 20:04:40.847758 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.847731    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-workload-socket" pod "44eb96c1-778e-433f-ae45-3a42beed5dac" (UID: "44eb96c1-778e-433f-ae45-3a42beed5dac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:40.847896 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.847835    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44eb96c1-778e-433f-ae45-3a42beed5dac-istiod-ca-cert" pod "44eb96c1-778e-433f-ae45-3a42beed5dac" (UID: "44eb96c1-778e-433f-ae45-3a42beed5dac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:40.848487 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.848460    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:40.848592 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.848493    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:40.848592 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.848509    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eb96c1-778e-433f-ae45-3a42beed5dac-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:40.848592 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.848523    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:40.848592 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.848538    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:40.849297 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.849271    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44eb96c1-778e-433f-ae45-3a42beed5dac-kube-api-access-w4fgh" pod "44eb96c1-778e-433f-ae45-3a42beed5dac" (UID: "44eb96c1-778e-433f-ae45-3a42beed5dac"). InnerVolumeSpecName "kube-api-access-w4fgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:40.849436 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.849412    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-envoy" pod "44eb96c1-778e-433f-ae45-3a42beed5dac" (UID: "44eb96c1-778e-433f-ae45-3a42beed5dac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:40.850134 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.850097    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-podinfo" pod "44eb96c1-778e-433f-ae45-3a42beed5dac" (UID: "44eb96c1-778e-433f-ae45-3a42beed5dac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:04:40.850550 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.850529    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-token" pod "44eb96c1-778e-433f-ae45-3a42beed5dac" (UID: "44eb96c1-778e-433f-ae45-3a42beed5dac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:40.950542 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.950508    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:40.950542 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.950541    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:40.950956 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.950556    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eb96c1-778e-433f-ae45-3a42beed5dac-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:40.950956 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:40.950566    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4fgh\" (UniqueName: \"kubernetes.io/projected/44eb96c1-778e-433f-ae45-3a42beed5dac-kube-api-access-w4fgh\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:41.000312 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:41.000230    4823 generic.go:356] "Generic (PLEG): container finished" podID="44eb96c1-778e-433f-ae45-3a42beed5dac" containerID="671e1f1c862980d941a1747b1d9975fcf00760e7b94e103e1af167f3e9a1ad96" exitCode=0
Oct 30 20:04:41.000312 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:41.000277    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"
Oct 30 20:04:41.000524 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:41.000307    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc" event={"ID":"44eb96c1-778e-433f-ae45-3a42beed5dac","Type":"ContainerDied","Data":"671e1f1c862980d941a1747b1d9975fcf00760e7b94e103e1af167f3e9a1ad96"}
Oct 30 20:04:41.000524 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:41.000369    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc" event={"ID":"44eb96c1-778e-433f-ae45-3a42beed5dac","Type":"ContainerDied","Data":"b0246e5bc68241fbec8647a1ebf7213ce318c6e98c6aa0dab0cf2f0090cd9986"}
Oct 30 20:04:41.000524 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:41.000404    4823 scope.go:122] "RemoveContainer" containerID="671e1f1c862980d941a1747b1d9975fcf00760e7b94e103e1af167f3e9a1ad96"
Oct 30 20:04:41.013782 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:41.013759    4823 scope.go:122] "RemoveContainer" containerID="671e1f1c862980d941a1747b1d9975fcf00760e7b94e103e1af167f3e9a1ad96"
Oct 30 20:04:41.014388 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:04:41.014364    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"671e1f1c862980d941a1747b1d9975fcf00760e7b94e103e1af167f3e9a1ad96\": container with ID starting with 671e1f1c862980d941a1747b1d9975fcf00760e7b94e103e1af167f3e9a1ad96 not found: ID does not exist" containerID="671e1f1c862980d941a1747b1d9975fcf00760e7b94e103e1af167f3e9a1ad96"
Oct 30 20:04:41.014499 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:41.014395    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"671e1f1c862980d941a1747b1d9975fcf00760e7b94e103e1af167f3e9a1ad96"} err="failed to get container status \"671e1f1c862980d941a1747b1d9975fcf00760e7b94e103e1af167f3e9a1ad96\": rpc error: code = NotFound desc = could not find container \"671e1f1c862980d941a1747b1d9975fcf00760e7b94e103e1af167f3e9a1ad96\": container with ID starting with 671e1f1c862980d941a1747b1d9975fcf00760e7b94e103e1af167f3e9a1ad96 not found: ID does not exist"
Oct 30 20:04:41.073255 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:41.073219    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"]
Oct 30 20:04:41.103463 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:41.103427    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-8w8jc"]
Oct 30 20:04:42.282388 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:42.282352    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44eb96c1-778e-433f-ae45-3a42beed5dac" path="/var/lib/kubelet/pods/44eb96c1-778e-433f-ae45-3a42beed5dac/volumes"
Oct 30 20:04:51.120196 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:04:51.120160    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:05:01.361444 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:01.360485    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:05:01.381300 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:01.381267    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:05:02.019952 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:05:02.019912    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:05:06.625044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:06.625006    4823 generic.go:356] "Generic (PLEG): container finished" podID="9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" containerID="7888a3364ad8d80ae22a582ee90f584d2243846afbf27550d4277f74b76ed584" exitCode=0
Oct 30 20:05:06.625565 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:06.625045    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d" event={"ID":"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15","Type":"ContainerDied","Data":"7888a3364ad8d80ae22a582ee90f584d2243846afbf27550d4277f74b76ed584"}
Oct 30 20:05:06.625565 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:06.625541    4823 scope.go:122] "RemoveContainer" containerID="7888a3364ad8d80ae22a582ee90f584d2243846afbf27550d4277f74b76ed584"
Oct 30 20:05:06.627049 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:06.627021    4823 generic.go:356] "Generic (PLEG): container finished" podID="835abdec-0719-4f64-b7e8-4e336b6f12bf" containerID="b8cb4e161b5701d775fc397a00ff0cc8ed987ace4598c454879f20e69f1c0631" exitCode=0
Oct 30 20:05:06.627170 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:06.627089    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj" event={"ID":"835abdec-0719-4f64-b7e8-4e336b6f12bf","Type":"ContainerDied","Data":"b8cb4e161b5701d775fc397a00ff0cc8ed987ace4598c454879f20e69f1c0631"}
Oct 30 20:05:06.627516 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:06.627500    4823 scope.go:122] "RemoveContainer" containerID="b8cb4e161b5701d775fc397a00ff0cc8ed987ace4598c454879f20e69f1c0631"
Oct 30 20:05:06.627674 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:06.627646    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:05:06.627759 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:06.627739    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:05:06.627847 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:06.627819    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:05:06.629687 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:06.629653    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:05:06.629776 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:06.629735    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:05:06.629820 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:06.629782    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:05:07.638928 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:07.638840    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj" event={"ID":"835abdec-0719-4f64-b7e8-4e336b6f12bf","Type":"ContainerStarted","Data":"3d33cd15521c4ef1ab20786f1104b05523523a7a275ea9567ce794c082fe7900"}
Oct 30 20:05:07.642063 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:07.642012    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d" event={"ID":"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15","Type":"ContainerStarted","Data":"a638832017626a6a1926db39045c72c992f5b56f42dc5873123116bdbccd2d59"}
Oct 30 20:05:08.301185 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:08.301148    4823 patch_prober.go:29] interesting pod/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:05:08.301495 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:08.301453    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d" podUID="9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 20:05:08.318585 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:08.318558    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:05:08.445775 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:08.445744    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:05:08.446124 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:08.446098    4823 patch_prober.go:29] interesting pod/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 20:05:08.446292 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:08.446161    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj" podUID="835abdec-0719-4f64-b7e8-4e336b6f12bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 20:05:09.339488 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:09.339005    4823 patch_prober.go:29] interesting pod/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:05:09.339488 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:09.339094    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d" podUID="9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 20:05:09.456732 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:09.456684    4823 patch_prober.go:29] interesting pod/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 20:05:09.456950 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:09.456748    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj" podUID="835abdec-0719-4f64-b7e8-4e336b6f12bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 20:05:10.324934 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:10.324046    4823 patch_prober.go:29] interesting pod/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:05:10.324934 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:10.324100    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d" podUID="9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 20:05:10.445391 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:10.445353    4823 patch_prober.go:29] interesting pod/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 20:05:10.445908 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:10.445407    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj" podUID="835abdec-0719-4f64-b7e8-4e336b6f12bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 20:05:11.303296 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:11.303255    4823 patch_prober.go:29] interesting pod/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:05:11.303697 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:11.303662    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d" podUID="9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 20:05:11.445929 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:11.445889    4823 patch_prober.go:29] interesting pod/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 20:05:11.446423 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:11.445958    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj" podUID="835abdec-0719-4f64-b7e8-4e336b6f12bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 20:05:12.305830 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:12.305286    4823 patch_prober.go:29] interesting pod/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:05:12.305830 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:12.305345    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d" podUID="9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 20:05:12.445931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:12.445889    4823 patch_prober.go:29] interesting pod/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 20:05:12.446204 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:12.446162    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj" podUID="835abdec-0719-4f64-b7e8-4e336b6f12bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 20:05:13.306593 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:13.306458    4823 patch_prober.go:29] interesting pod/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:05:13.307017 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:13.306984    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d" podUID="9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 20:05:13.445517 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:13.445478    4823 patch_prober.go:29] interesting pod/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 20:05:13.445701 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:13.445549    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj" podUID="835abdec-0719-4f64-b7e8-4e336b6f12bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 20:05:13.517511 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:13.517469    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"]
Oct 30 20:05:13.518055 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:13.517735    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf" podUID="86bda8bd-f59d-4f99-aafb-cccfbf84384c" containerName="istio-proxy" containerID="cri-o://1308912743c5704062e89d13177e28a8948585cf59d0bf0dd0b843991fce079c" gracePeriod=30
Oct 30 20:05:14.073155 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:14.073117    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:05:14.306139 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:14.306112    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:05:14.306626 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:14.306608    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:05:14.307447 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:14.307428    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:05:14.450464 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:14.450435    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:05:14.885293 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:14.885259    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:05:18.824641 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.824610    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 20:05:18.918542 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.918505    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-data\") pod \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") "
Oct 30 20:05:18.918770 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.918578    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86bda8bd-f59d-4f99-aafb-cccfbf84384c-kube-api-access-2m427\" (UniqueName: \"kubernetes.io/projected/86bda8bd-f59d-4f99-aafb-cccfbf84384c-kube-api-access-2m427\") pod \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") "
Oct 30 20:05:18.918770 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.918657    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-workload-certs\") pod \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") "
Oct 30 20:05:18.918770 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.918727    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-credential-socket\") pod \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") "
Oct 30 20:05:18.919038 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.918779    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-data" pod "86bda8bd-f59d-4f99-aafb-cccfbf84384c" (UID: "86bda8bd-f59d-4f99-aafb-cccfbf84384c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:18.919038 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.918921    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-token\" (UniqueName: \"kubernetes.io/projected/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-token\") pod \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") "
Oct 30 20:05:18.919154 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.919094    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-envoy\") pod \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") "
Oct 30 20:05:18.919154 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.919132    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-workload-socket\") pod \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") "
Oct 30 20:05:18.919274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.919169    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istiod-ca-cert\") pod \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") "
Oct 30 20:05:18.919274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.919239    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-podinfo\") pod \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\" (UID: \"86bda8bd-f59d-4f99-aafb-cccfbf84384c\") "
Oct 30 20:05:18.919648 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.919622    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-credential-socket" pod "86bda8bd-f59d-4f99-aafb-cccfbf84384c" (UID: "86bda8bd-f59d-4f99-aafb-cccfbf84384c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:18.919808 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.919636    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-workload-certs" pod "86bda8bd-f59d-4f99-aafb-cccfbf84384c" (UID: "86bda8bd-f59d-4f99-aafb-cccfbf84384c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:18.919923 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.919823    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-workload-socket" pod "86bda8bd-f59d-4f99-aafb-cccfbf84384c" (UID: "86bda8bd-f59d-4f99-aafb-cccfbf84384c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:18.920209 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.920182    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:18.920311 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.920215    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:18.920311 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.920230    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:18.920311 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.920244    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:18.920625 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.920602    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istiod-ca-cert" pod "86bda8bd-f59d-4f99-aafb-cccfbf84384c" (UID: "86bda8bd-f59d-4f99-aafb-cccfbf84384c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:18.923416 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.923330    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-token" pod "86bda8bd-f59d-4f99-aafb-cccfbf84384c" (UID: "86bda8bd-f59d-4f99-aafb-cccfbf84384c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:18.923535 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.923471    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86bda8bd-f59d-4f99-aafb-cccfbf84384c-kube-api-access-2m427" pod "86bda8bd-f59d-4f99-aafb-cccfbf84384c" (UID: "86bda8bd-f59d-4f99-aafb-cccfbf84384c"). InnerVolumeSpecName "kube-api-access-2m427". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:18.924553 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.924344    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-podinfo" pod "86bda8bd-f59d-4f99-aafb-cccfbf84384c" (UID: "86bda8bd-f59d-4f99-aafb-cccfbf84384c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:05:18.925390 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.925005    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-envoy" pod "86bda8bd-f59d-4f99-aafb-cccfbf84384c" (UID: "86bda8bd-f59d-4f99-aafb-cccfbf84384c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:18.928569 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.928541    4823 generic.go:356] "Generic (PLEG): container finished" podID="86bda8bd-f59d-4f99-aafb-cccfbf84384c" containerID="1308912743c5704062e89d13177e28a8948585cf59d0bf0dd0b843991fce079c" exitCode=0
Oct 30 20:05:18.928674 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.928605    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"
Oct 30 20:05:18.928674 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.928627    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf" event={"ID":"86bda8bd-f59d-4f99-aafb-cccfbf84384c","Type":"ContainerDied","Data":"1308912743c5704062e89d13177e28a8948585cf59d0bf0dd0b843991fce079c"}
Oct 30 20:05:18.928800 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.928694    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf" event={"ID":"86bda8bd-f59d-4f99-aafb-cccfbf84384c","Type":"ContainerDied","Data":"cc3352aa1a958e979fae7349a81e9c5a61601ffab1acbb5ef99c1c91b7298c38"}
Oct 30 20:05:18.928800 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.928725    4823 scope.go:122] "RemoveContainer" containerID="1308912743c5704062e89d13177e28a8948585cf59d0bf0dd0b843991fce079c"
Oct 30 20:05:18.968003 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.967981    4823 scope.go:122] "RemoveContainer" containerID="1308912743c5704062e89d13177e28a8948585cf59d0bf0dd0b843991fce079c"
Oct 30 20:05:18.968319 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:05:18.968299    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1308912743c5704062e89d13177e28a8948585cf59d0bf0dd0b843991fce079c\": container with ID starting with 1308912743c5704062e89d13177e28a8948585cf59d0bf0dd0b843991fce079c not found: ID does not exist" containerID="1308912743c5704062e89d13177e28a8948585cf59d0bf0dd0b843991fce079c"
Oct 30 20:05:18.968370 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.968332    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1308912743c5704062e89d13177e28a8948585cf59d0bf0dd0b843991fce079c"} err="failed to get container status \"1308912743c5704062e89d13177e28a8948585cf59d0bf0dd0b843991fce079c\": rpc error: code = NotFound desc = could not find container \"1308912743c5704062e89d13177e28a8948585cf59d0bf0dd0b843991fce079c\": container with ID starting with 1308912743c5704062e89d13177e28a8948585cf59d0bf0dd0b843991fce079c not found: ID does not exist"
Oct 30 20:05:18.976879 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.976837    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"]
Oct 30 20:05:18.982708 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:18.982681    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-lvhnf"]
Oct 30 20:05:19.052112 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:19.050185    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:19.052112 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:19.050227    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:19.052112 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:19.050243    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:19.052112 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:19.050258    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m427\" (UniqueName: \"kubernetes.io/projected/86bda8bd-f59d-4f99-aafb-cccfbf84384c-kube-api-access-2m427\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:19.052112 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:19.050273    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bda8bd-f59d-4f99-aafb-cccfbf84384c-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:20.282675 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:20.282641    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86bda8bd-f59d-4f99-aafb-cccfbf84384c" path="/var/lib/kubelet/pods/86bda8bd-f59d-4f99-aafb-cccfbf84384c/volumes"
Oct 30 20:05:24.317158 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.317121    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"]
Oct 30 20:05:24.318409 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.318371    4823 cpu_manager.go:414] "containerMap: removing container" podUID="118587b4-0b24-4a51-b462-68bce9602a0a" containerName="istio-proxy"
Oct 30 20:05:24.318647 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.318614    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118587b4-0b24-4a51-b462-68bce9602a0a" containerName="istio-proxy"
Oct 30 20:05:24.319174 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.318677    4823 cpu_manager.go:414] "containerMap: removing container" podUID="44eb96c1-778e-433f-ae45-3a42beed5dac" containerName="istio-proxy"
Oct 30 20:05:24.319174 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.318687    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44eb96c1-778e-433f-ae45-3a42beed5dac" containerName="istio-proxy"
Oct 30 20:05:24.319174 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.318758    4823 cpu_manager.go:414] "containerMap: removing container" podUID="86bda8bd-f59d-4f99-aafb-cccfbf84384c" containerName="istio-proxy"
Oct 30 20:05:24.319174 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.318766    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86bda8bd-f59d-4f99-aafb-cccfbf84384c" containerName="istio-proxy"
Oct 30 20:05:24.319174 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.318781    4823 cpu_manager.go:414] "containerMap: removing container" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy"
Oct 30 20:05:24.319174 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.318789    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy"
Oct 30 20:05:24.319960 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.319937    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4bdb13-b5bb-4692-bbff-769a263dcffa" containerName="istio-proxy"
Oct 30 20:05:24.320160 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.320140    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="118587b4-0b24-4a51-b462-68bce9602a0a" containerName="istio-proxy"
Oct 30 20:05:24.320259 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.320169    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="44eb96c1-778e-433f-ae45-3a42beed5dac" containerName="istio-proxy"
Oct 30 20:05:24.320259 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.320195    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="86bda8bd-f59d-4f99-aafb-cccfbf84384c" containerName="istio-proxy"
Oct 30 20:05:24.323918 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.323851    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:05:24.336148 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.333906    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-19f2ee-serving-cert\""
Oct 30 20:05:24.339151 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.338891    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"]
Oct 30 20:05:24.520890 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.491667    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k67cc\" (UniqueName: \"kubernetes.io/projected/9dfba44f-22b1-4336-8938-b29d8478811f-kube-api-access-k67cc\") pod \"maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd\" (UID: \"9dfba44f-22b1-4336-8938-b29d8478811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:05:24.521327 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.521289    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dfba44f-22b1-4336-8938-b29d8478811f-maas-api-tls\") pod \"maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd\" (UID: \"9dfba44f-22b1-4336-8938-b29d8478811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:05:24.528895 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.521512    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dfba44f-22b1-4336-8938-b29d8478811f-metrics-certs\") pod \"maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd\" (UID: \"9dfba44f-22b1-4336-8938-b29d8478811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:05:24.529199 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.529164    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfba44f-22b1-4336-8938-b29d8478811f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd\" (UID: \"9dfba44f-22b1-4336-8938-b29d8478811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:05:24.631479 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.631432    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k67cc\" (UniqueName: \"kubernetes.io/projected/9dfba44f-22b1-4336-8938-b29d8478811f-kube-api-access-k67cc\") pod \"maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd\" (UID: \"9dfba44f-22b1-4336-8938-b29d8478811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:05:24.631726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.631562    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dfba44f-22b1-4336-8938-b29d8478811f-maas-api-tls\") pod \"maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd\" (UID: \"9dfba44f-22b1-4336-8938-b29d8478811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:05:24.631726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.631615    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dfba44f-22b1-4336-8938-b29d8478811f-metrics-certs\") pod \"maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd\" (UID: \"9dfba44f-22b1-4336-8938-b29d8478811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:05:24.631726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.631641    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfba44f-22b1-4336-8938-b29d8478811f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd\" (UID: \"9dfba44f-22b1-4336-8938-b29d8478811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:05:24.632701 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.632658    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfba44f-22b1-4336-8938-b29d8478811f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd\" (UID: \"9dfba44f-22b1-4336-8938-b29d8478811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:05:24.635428 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.635403    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dfba44f-22b1-4336-8938-b29d8478811f-maas-api-tls\") pod \"maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd\" (UID: \"9dfba44f-22b1-4336-8938-b29d8478811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:05:24.635582 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.635554    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dfba44f-22b1-4336-8938-b29d8478811f-metrics-certs\") pod \"maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd\" (UID: \"9dfba44f-22b1-4336-8938-b29d8478811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:05:24.646137 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.646111    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k67cc\" (UniqueName: \"kubernetes.io/projected/9dfba44f-22b1-4336-8938-b29d8478811f-kube-api-access-k67cc\") pod \"maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd\" (UID: \"9dfba44f-22b1-4336-8938-b29d8478811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:05:24.646983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:24.646957    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:05:25.083507 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:25.083469    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"]
Oct 30 20:05:25.092735 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:05:25.092686    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dfba44f_22b1_4336_8938_b29d8478811f.slice/crio-f6384ee1353922c61815000810965e0d4cd691b687e579c7a4689833340d16ca WatchSource:0}: Error finding container f6384ee1353922c61815000810965e0d4cd691b687e579c7a4689833340d16ca: Status 404 returned error can't find the container with id f6384ee1353922c61815000810965e0d4cd691b687e579c7a4689833340d16ca
Oct 30 20:05:26.100951 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:26.100839    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd" event={"ID":"9dfba44f-22b1-4336-8938-b29d8478811f","Type":"ContainerStarted","Data":"d87e9b6a7434c574b39820d760b36e4c8a256411b8d8094015e556907f00cce7"}
Oct 30 20:05:26.100951 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:26.100937    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd" event={"ID":"9dfba44f-22b1-4336-8938-b29d8478811f","Type":"ContainerStarted","Data":"f6384ee1353922c61815000810965e0d4cd691b687e579c7a4689833340d16ca"}
Oct 30 20:05:26.101420 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:26.100963    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:05:26.121452 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:26.121392    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd" podStartSLOduration=1.5778686880000001 podStartE2EDuration="2.121373272s" totalImagesPullingTime="543.504584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:26.120300103 +0000 UTC m=+2126.012914426" watchObservedRunningTime="2026-10-30 20:05:26.121373272 +0000 UTC m=+2126.013987598"
Oct 30 20:05:27.935443 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:27.935407    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"]
Oct 30 20:05:27.940082 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:27.940061    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:27.950295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:27.949852    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f246b-openshift-default-dockercfg-nb8cn\""
Oct 30 20:05:27.953221 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:27.953179    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"]
Oct 30 20:05:28.024693 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.024648    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-workload-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.024905 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.024709    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-data\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.024975 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.024944    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqgd9\" (UniqueName: \"kubernetes.io/projected/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-kube-api-access-jqgd9\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.025031 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.025004    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istiod-ca-cert\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.025111 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.025091    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-envoy\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.025169 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.025154    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-token\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.025287 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.025270    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-podinfo\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.025417 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.025397    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-workload-certs\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.025532 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.025518    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-credential-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.126712 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.126680    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istiod-ca-cert\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.126900 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.126740    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-envoy\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.126900 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.126783    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-token\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.126900 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.126877    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-podinfo\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.127025 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.126970    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-workload-certs\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.127163 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.127073    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-credential-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.127163 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.127147    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-workload-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.127276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.127189    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-data\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.127354 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.127303    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqgd9\" (UniqueName: \"kubernetes.io/projected/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-kube-api-access-jqgd9\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.127408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.127373    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-workload-certs\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.127460 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.127446    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istiod-ca-cert\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.127705 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.127681    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-data\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.127787 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.127697    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-workload-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.127787 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.127762    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-credential-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.129378 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.129354    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-envoy\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.129569 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.129552    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-podinfo\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.139682 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.139655    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-token\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.139811 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.139792    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqgd9\" (UniqueName: \"kubernetes.io/projected/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-kube-api-access-jqgd9\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.258497 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.258415    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:28.567925 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:05:28.567888    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0d0374_f31f_42dc_a1f7_9fc3a4d845e3.slice/crio-f6c2aed8505c97870eb77792193111f839907344b2f575d2ba0bf133d3e7811b WatchSource:0}: Error finding container f6c2aed8505c97870eb77792193111f839907344b2f575d2ba0bf133d3e7811b: Status 404 returned error can't find the container with id f6c2aed8505c97870eb77792193111f839907344b2f575d2ba0bf133d3e7811b
Oct 30 20:05:28.570207 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.570154    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"]
Oct 30 20:05:28.572669 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.572493    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:05:28.572669 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.572578    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:05:28.572669 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:28.572612    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:05:29.129698 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.129655    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf" event={"ID":"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3","Type":"ContainerStarted","Data":"ddc86e55ca9e85af299f3cebbd5adfd4db241e0fcca24435b0169ac82ac1f7bc"}
Oct 30 20:05:29.130149 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.129768    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf" event={"ID":"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3","Type":"ContainerStarted","Data":"f6c2aed8505c97870eb77792193111f839907344b2f575d2ba0bf133d3e7811b"}
Oct 30 20:05:29.157661 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.157587    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf" podStartSLOduration=2.157571962 podStartE2EDuration="2.157571962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:29.152247673 +0000 UTC m=+2129.044861997" watchObservedRunningTime="2026-10-30 20:05:29.157571962 +0000 UTC m=+2129.050186286"
Oct 30 20:05:29.272923 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.265133    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:29.275142 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.274995    4823 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 20:05:29.275142 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.275059    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf" podUID="2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 20:05:29.289011 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.288018    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"]
Oct 30 20:05:29.292466 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.292442    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:29.307987 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.302460    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-55d2e0-serving-cert\""
Oct 30 20:05:29.327606 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.326402    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"]
Oct 30 20:05:29.382962 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.378344    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:29.382962 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.378399    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6p9d\" (UniqueName: \"kubernetes.io/projected/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-kube-api-access-j6p9d\") pod \"maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:29.382962 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.378517    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-metrics-certs\") pod \"maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:29.382962 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.379044    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:29.498895 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.492562    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:29.498895 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.492891    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:29.498895 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.492929    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6p9d\" (UniqueName: \"kubernetes.io/projected/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-kube-api-access-j6p9d\") pod \"maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:29.498895 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.493071    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-metrics-certs\") pod \"maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:29.530117 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.523768    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:29.530117 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:05:29.523966    4823 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-serving-cert: secret "maas-api-e2e-inf-b-55d2e0-serving-cert" not found
Oct 30 20:05:29.530117 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:05:29.524036    4823 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-maas-api-tls podName:38b2ae10-5954-4364-83f3-7e79ee7ad4f9 nodeName:}" failed. No retries permitted until 2026-10-30 20:05:30.024014882 +0000 UTC m=+2129.916629190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-maas-api-tls") pod "maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg" (UID: "38b2ae10-5954-4364-83f3-7e79ee7ad4f9") : secret "maas-api-e2e-inf-b-55d2e0-serving-cert" not found
Oct 30 20:05:29.548712 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.548670    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6p9d\" (UniqueName: \"kubernetes.io/projected/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-kube-api-access-j6p9d\") pod \"maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:29.565954 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:29.553799    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-metrics-certs\") pod \"maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:30.038884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:30.027723    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:30.077837 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:30.073722    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:30.223903 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:30.223170    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:30.310154 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:30.310058    4823 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 20:05:30.310154 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:30.310135    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf" podUID="2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 20:05:30.607609 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:30.607552    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"]
Oct 30 20:05:30.607784 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:05:30.607676    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b2ae10_5954_4364_83f3_7e79ee7ad4f9.slice/crio-71fb9f2871ee5b9d649dd01ac41a2e5482c7d4d5dcfaf0a840d5b88c8e661910 WatchSource:0}: Error finding container 71fb9f2871ee5b9d649dd01ac41a2e5482c7d4d5dcfaf0a840d5b88c8e661910: Status 404 returned error can't find the container with id 71fb9f2871ee5b9d649dd01ac41a2e5482c7d4d5dcfaf0a840d5b88c8e661910
Oct 30 20:05:30.619535 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:30.609777    4823 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:05:31.202812 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:31.202773    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg" event={"ID":"38b2ae10-5954-4364-83f3-7e79ee7ad4f9","Type":"ContainerStarted","Data":"71fb9f2871ee5b9d649dd01ac41a2e5482c7d4d5dcfaf0a840d5b88c8e661910"}
Oct 30 20:05:31.268533 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:31.262280    4823 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 20:05:31.268533 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:31.262345    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf" podUID="2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 20:05:32.017498 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:05:32.017459    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:05:32.115322 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:32.115286    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:05:32.215700 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:32.215667    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg" event={"ID":"38b2ae10-5954-4364-83f3-7e79ee7ad4f9","Type":"ContainerStarted","Data":"5d995d5ef1376be27eed251096b54dfe00b662d5d7f7c2695b56c221dd7bbba6"}
Oct 30 20:05:32.215926 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:32.215850    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:32.240152 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:32.240103    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg" podStartSLOduration=2.757945153 podStartE2EDuration="3.240089303s" totalImagesPullingTime="482.14415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:32.235477306 +0000 UTC m=+2132.128091630" watchObservedRunningTime="2026-10-30 20:05:32.240089303 +0000 UTC m=+2132.132703627"
Oct 30 20:05:32.277945 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:32.277658    4823 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 20:05:32.277945 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:32.277731    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf" podUID="2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 20:05:33.275886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:33.275318    4823 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 20:05:33.275886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:33.275395    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf" podUID="2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 20:05:34.277263 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:34.274278    4823 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Oct 30 20:05:34.277263 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:34.274340    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf" podUID="2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Oct 30 20:05:35.354365 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:35.354331    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:35.355052 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:35.355028    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:35.356184 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:35.356161    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:05:38.230888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:38.230837    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:05:39.003357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.003319    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"]
Oct 30 20:05:39.010521 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.010493    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:05:39.023574 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.023541    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"]
Oct 30 20:05:39.037304 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.037279    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6986a4-serving-cert\""
Oct 30 20:05:39.109953 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.109921    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dzbj\" (UniqueName: \"kubernetes.io/projected/b00c23a6-46e6-4db7-83c0-f8063c327df4-kube-api-access-5dzbj\") pod \"maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd\" (UID: \"b00c23a6-46e6-4db7-83c0-f8063c327df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:05:39.110258 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.110236    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b00c23a6-46e6-4db7-83c0-f8063c327df4-metrics-certs\") pod \"maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd\" (UID: \"b00c23a6-46e6-4db7-83c0-f8063c327df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:05:39.110372 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.110355    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00c23a6-46e6-4db7-83c0-f8063c327df4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd\" (UID: \"b00c23a6-46e6-4db7-83c0-f8063c327df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:05:39.110538 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.110510    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b00c23a6-46e6-4db7-83c0-f8063c327df4-maas-api-tls\") pod \"maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd\" (UID: \"b00c23a6-46e6-4db7-83c0-f8063c327df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:05:39.213335 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.213293    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b00c23a6-46e6-4db7-83c0-f8063c327df4-maas-api-tls\") pod \"maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd\" (UID: \"b00c23a6-46e6-4db7-83c0-f8063c327df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:05:39.219605 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.219571    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dzbj\" (UniqueName: \"kubernetes.io/projected/b00c23a6-46e6-4db7-83c0-f8063c327df4-kube-api-access-5dzbj\") pod \"maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd\" (UID: \"b00c23a6-46e6-4db7-83c0-f8063c327df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:05:39.237997 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.220263    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b00c23a6-46e6-4db7-83c0-f8063c327df4-metrics-certs\") pod \"maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd\" (UID: \"b00c23a6-46e6-4db7-83c0-f8063c327df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:05:39.237997 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.220408    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00c23a6-46e6-4db7-83c0-f8063c327df4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd\" (UID: \"b00c23a6-46e6-4db7-83c0-f8063c327df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:05:39.237997 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.221486    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00c23a6-46e6-4db7-83c0-f8063c327df4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd\" (UID: \"b00c23a6-46e6-4db7-83c0-f8063c327df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:05:39.245314 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.245280    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b00c23a6-46e6-4db7-83c0-f8063c327df4-maas-api-tls\") pod \"maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd\" (UID: \"b00c23a6-46e6-4db7-83c0-f8063c327df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:05:39.254058 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.253938    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dzbj\" (UniqueName: \"kubernetes.io/projected/b00c23a6-46e6-4db7-83c0-f8063c327df4-kube-api-access-5dzbj\") pod \"maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd\" (UID: \"b00c23a6-46e6-4db7-83c0-f8063c327df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:05:39.284150 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.282950    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b00c23a6-46e6-4db7-83c0-f8063c327df4-metrics-certs\") pod \"maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd\" (UID: \"b00c23a6-46e6-4db7-83c0-f8063c327df4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:05:39.348798 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.348763    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:05:39.732776 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:39.732713    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"]
Oct 30 20:05:39.738019 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:05:39.735945    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00c23a6_46e6_4db7_83c0_f8063c327df4.slice/crio-74e9901131bb02b8f8c8b6f2e2a72635ec3689213a06961b5c5e5eaf5f88f7c7 WatchSource:0}: Error finding container 74e9901131bb02b8f8c8b6f2e2a72635ec3689213a06961b5c5e5eaf5f88f7c7: Status 404 returned error can't find the container with id 74e9901131bb02b8f8c8b6f2e2a72635ec3689213a06961b5c5e5eaf5f88f7c7
Oct 30 20:05:40.323928 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:40.323893    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd" event={"ID":"b00c23a6-46e6-4db7-83c0-f8063c327df4","Type":"ContainerStarted","Data":"74e9901131bb02b8f8c8b6f2e2a72635ec3689213a06961b5c5e5eaf5f88f7c7"}
Oct 30 20:05:41.335726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:41.335688    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd" event={"ID":"b00c23a6-46e6-4db7-83c0-f8063c327df4","Type":"ContainerStarted","Data":"28fc5e0ce7f5b4068c57377001379f15d25303cceb84b2be8dec5b14d8e6595f"}
Oct 30 20:05:41.336565 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:41.335760    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:05:41.359480 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:41.359423    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd" podStartSLOduration=2.839814274 podStartE2EDuration="3.35940993s" totalImagesPullingTime="519.595656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:41.354644798 +0000 UTC m=+2141.247259123" watchObservedRunningTime="2026-10-30 20:05:41.35940993 +0000 UTC m=+2141.252024253"
Oct 30 20:05:46.895227 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:46.895195    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:05:47.350191 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:47.350110    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:05:57.032362 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:57.032322    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"]
Oct 30 20:05:57.036821 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:57.036796    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-060bf0e1-ghq2f"]
Oct 30 20:05:58.372033 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:05:58.371994    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8ffe44-35d9-40c2-b767-b5610445757b" path="/var/lib/kubelet/pods/fd8ffe44-35d9-40c2-b767-b5610445757b/volumes"
Oct 30 20:06:00.760999 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:00.760967    4823 generic.go:356] "Generic (PLEG): container finished" podID="04c51f58-c2b5-4cef-858e-512b42dba670" containerID="170d06bcbe3fa01a2f5ba04610511898c74a189174ec323cbbfb5e15f5f3eb49" exitCode=0
Oct 30 20:06:00.761409 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:00.761037    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq" event={"ID":"04c51f58-c2b5-4cef-858e-512b42dba670","Type":"ContainerDied","Data":"170d06bcbe3fa01a2f5ba04610511898c74a189174ec323cbbfb5e15f5f3eb49"}
Oct 30 20:06:00.761551 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:00.761534    4823 scope.go:122] "RemoveContainer" containerID="170d06bcbe3fa01a2f5ba04610511898c74a189174ec323cbbfb5e15f5f3eb49"
Oct 30 20:06:00.763658 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:00.763623    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:06:00.763763 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:00.763699    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:06:00.763763 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:00.763733    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:06:01.832910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:01.832759    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq" event={"ID":"04c51f58-c2b5-4cef-858e-512b42dba670","Type":"ContainerStarted","Data":"0d9aab1ff4d48482600f093963ef51da5b0d9e6dcf8cdc39962226d28cc60033"}
Oct 30 20:06:02.016621 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:06:02.016587    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:06:02.319124 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:02.319087    4823 scope.go:122] "RemoveContainer" containerID="1260f8f4a8626e0182c9609d4d70cb0cd0a8c0c90cc889a7603d24e707ce681f"
Oct 30 20:06:02.560271 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:02.560185    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:06:02.573933 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:02.568912    4823 patch_prober.go:29] interesting pod/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 20:06:02.573933 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:02.568978    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq" podUID="04c51f58-c2b5-4cef-858e-512b42dba670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 20:06:03.577091 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:03.577051    4823 patch_prober.go:29] interesting pod/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 20:06:03.577715 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:03.577681    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq" podUID="04c51f58-c2b5-4cef-858e-512b42dba670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 20:06:04.561256 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:04.561219    4823 patch_prober.go:29] interesting pod/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 20:06:04.561547 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:04.561508    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq" podUID="04c51f58-c2b5-4cef-858e-512b42dba670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 20:06:05.563392 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:05.563353    4823 patch_prober.go:29] interesting pod/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 20:06:05.564081 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:05.564047    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq" podUID="04c51f58-c2b5-4cef-858e-512b42dba670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 20:06:06.558370 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:06.558334    4823 patch_prober.go:29] interesting pod/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 20:06:06.558658 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:06.558626    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq" podUID="04c51f58-c2b5-4cef-858e-512b42dba670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 20:06:06.994267 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:06.994230    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:06:07.724804 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:07.724761    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:06:07.726497 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:07.726456    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:06:11.531490 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.531453    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"]
Oct 30 20:06:11.538332 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.538311    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.541910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.541881    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-77b4df-openshift-default-dockercfg-47njs\""
Oct 30 20:06:11.548687 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.548667    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"]
Oct 30 20:06:11.635267 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.635220    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-data\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.635514 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.635297    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-credential-socket\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.635514 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.635461    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-workload-socket\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.635612 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.635573    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-podinfo\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.635661 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.635645    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4543ad-ea76-4398-b7f2-7d7a32132821-istiod-ca-cert\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.635797 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.635779    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4swls\" (UniqueName: \"kubernetes.io/projected/3c4543ad-ea76-4398-b7f2-7d7a32132821-kube-api-access-4swls\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.635855 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.635834    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-envoy\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.635925 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.635908    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-workload-certs\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.636175 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.636149    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-token\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.770886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.748764    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-podinfo\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.780550 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.780514    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4543ad-ea76-4398-b7f2-7d7a32132821-istiod-ca-cert\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.781003 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.780971    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4swls\" (UniqueName: \"kubernetes.io/projected/3c4543ad-ea76-4398-b7f2-7d7a32132821-kube-api-access-4swls\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.781223 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.781187    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-envoy\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.781408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.781390    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-workload-certs\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.781648 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.781598    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-token\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.781980 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.781939    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-data\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.782281 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.782237    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4543ad-ea76-4398-b7f2-7d7a32132821-istiod-ca-cert\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.783011 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.782976    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-credential-socket\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.783436 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.783410    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-credential-socket\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.783729 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.783456    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-data\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.783941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.783727    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-workload-certs\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.784018 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.783961    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-workload-socket\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.784632 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.784595    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-workload-socket\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.809713 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.806237    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-podinfo\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.819402 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.813700    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-token\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.826712 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.825077    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-envoy\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.826712 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.826403    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4swls\" (UniqueName: \"kubernetes.io/projected/3c4543ad-ea76-4398-b7f2-7d7a32132821-kube-api-access-4swls\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:11.857401 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:11.857368    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:12.185719 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:06:12.185683    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4543ad_ea76_4398_b7f2_7d7a32132821.slice/crio-a420ce8f09181eb46c69ecc6302d5a7eb1e663858452f864a0098c651728837e WatchSource:0}: Error finding container a420ce8f09181eb46c69ecc6302d5a7eb1e663858452f864a0098c651728837e: Status 404 returned error can't find the container with id a420ce8f09181eb46c69ecc6302d5a7eb1e663858452f864a0098c651728837e
Oct 30 20:06:12.185854 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:12.185840    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"]
Oct 30 20:06:12.188749 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:12.188719    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:06:12.188831 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:12.188783    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:06:12.188831 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:12.188809    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:06:13.073355 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:13.073308    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw" event={"ID":"3c4543ad-ea76-4398-b7f2-7d7a32132821","Type":"ContainerStarted","Data":"d6eb2da20b5918925e4f098d2183523084dc4ec1ad8a043136ca04ede720116a"}
Oct 30 20:06:13.073795 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:13.073373    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw" event={"ID":"3c4543ad-ea76-4398-b7f2-7d7a32132821","Type":"ContainerStarted","Data":"a420ce8f09181eb46c69ecc6302d5a7eb1e663858452f864a0098c651728837e"}
Oct 30 20:06:13.105837 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:13.105780    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw" podStartSLOduration=2.105766872 podStartE2EDuration="2.105766872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:13.103319073 +0000 UTC m=+2172.995933397" watchObservedRunningTime="2026-10-30 20:06:13.105766872 +0000 UTC m=+2172.998381195"
Oct 30 20:06:13.870887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:13.869222    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:13.877048 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:13.877007    4823 patch_prober.go:29] interesting pod/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 20:06:13.877262 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:13.877075    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw" podUID="3c4543ad-ea76-4398-b7f2-7d7a32132821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 20:06:14.862883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:14.860420    4823 patch_prober.go:29] interesting pod/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 20:06:14.862883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:14.860484    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw" podUID="3c4543ad-ea76-4398-b7f2-7d7a32132821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 20:06:15.876890 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:15.871279    4823 patch_prober.go:29] interesting pod/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 20:06:15.876890 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:15.871342    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw" podUID="3c4543ad-ea76-4398-b7f2-7d7a32132821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 20:06:16.877924 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:16.877014    4823 patch_prober.go:29] interesting pod/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 20:06:16.877924 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:16.877091    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw" podUID="3c4543ad-ea76-4398-b7f2-7d7a32132821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 20:06:17.877085 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:17.877048    4823 patch_prober.go:29] interesting pod/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 20:06:17.877295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:17.877163    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw" podUID="3c4543ad-ea76-4398-b7f2-7d7a32132821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 20:06:18.857872 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:18.857827    4823 patch_prober.go:29] interesting pod/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 20:06:18.858315 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:18.857893    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw" podUID="3c4543ad-ea76-4398-b7f2-7d7a32132821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 20:06:19.863974 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:19.863932    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:19.864466 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:19.864287    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:19.865091 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:19.865074    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:06:32.017480 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:06:32.017438    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:06:52.598646 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:52.598567    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:06:54.827931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:54.827628    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 20:06:54.992073 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:54.992010    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.214680 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:55.214135    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:06:55.697269 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:55.695975    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.717994 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:55.717946    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 30 20:06:55.735140 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:55.735082    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.792235 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:55.792190    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw" podUID="531e5023-637b-4a75-9846-8e53b962fc14" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 20:06:55.844885 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:55.840082    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.998982 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:55.993982    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:55.998982 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:55.994623    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 20:06:56.164523 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:56.161967    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 20:06:56.438894 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:06:56.438797    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 20:07:02.016694 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:07:02.016653    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:07:02.390410 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:02.390378    4823 scope.go:122] "RemoveContainer" containerID="46b16e1d73100bc60333796285b985f790a0af1c3c2e07849d86f4989b797ddf"
Oct 30 20:07:32.015902 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:07:32.015844    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:07:42.941522 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:42.941483    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"]
Oct 30 20:07:42.942073 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:42.941699    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws" podUID="091b9ee1-723f-44d7-bac5-629178dbd234" containerName="istio-proxy" containerID="cri-o://a8db6bb9b415b6b60261828b6ac44c80194095b9c245aed2712caf1acee58521" gracePeriod=30
Oct 30 20:07:48.261624 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.261594    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 20:07:48.321466 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.321432    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-credential-socket\") pod \"091b9ee1-723f-44d7-bac5-629178dbd234\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") "
Oct 30 20:07:48.321657 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.321477    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091b9ee1-723f-44d7-bac5-629178dbd234-istio-token\" (UniqueName: \"kubernetes.io/projected/091b9ee1-723f-44d7-bac5-629178dbd234-istio-token\") pod \"091b9ee1-723f-44d7-bac5-629178dbd234\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") "
Oct 30 20:07:48.321657 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.321608    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-istio-data\") pod \"091b9ee1-723f-44d7-bac5-629178dbd234\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") "
Oct 30 20:07:48.321775 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.321700    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-workload-certs\") pod \"091b9ee1-723f-44d7-bac5-629178dbd234\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") "
Oct 30 20:07:48.321775 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.321746    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-credential-socket" pod "091b9ee1-723f-44d7-bac5-629178dbd234" (UID: "091b9ee1-723f-44d7-bac5-629178dbd234"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:07:48.322002 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.321983    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091b9ee1-723f-44d7-bac5-629178dbd234-kube-api-access-fd8dv\" (UniqueName: \"kubernetes.io/projected/091b9ee1-723f-44d7-bac5-629178dbd234-kube-api-access-fd8dv\") pod \"091b9ee1-723f-44d7-bac5-629178dbd234\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") "
Oct 30 20:07:48.322114 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.321974    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-istio-data" pod "091b9ee1-723f-44d7-bac5-629178dbd234" (UID: "091b9ee1-723f-44d7-bac5-629178dbd234"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:07:48.322114 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.322049    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/091b9ee1-723f-44d7-bac5-629178dbd234-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091b9ee1-723f-44d7-bac5-629178dbd234-istio-podinfo\") pod \"091b9ee1-723f-44d7-bac5-629178dbd234\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") "
Oct 30 20:07:48.322230 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.322116    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/091b9ee1-723f-44d7-bac5-629178dbd234-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091b9ee1-723f-44d7-bac5-629178dbd234-istiod-ca-cert\") pod \"091b9ee1-723f-44d7-bac5-629178dbd234\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") "
Oct 30 20:07:48.322230 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.322049    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-workload-certs" pod "091b9ee1-723f-44d7-bac5-629178dbd234" (UID: "091b9ee1-723f-44d7-bac5-629178dbd234"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:07:48.322230 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.322211    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-istio-envoy\") pod \"091b9ee1-723f-44d7-bac5-629178dbd234\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") "
Oct 30 20:07:48.322396 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.322258    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-workload-socket\") pod \"091b9ee1-723f-44d7-bac5-629178dbd234\" (UID: \"091b9ee1-723f-44d7-bac5-629178dbd234\") "
Oct 30 20:07:48.322724 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.322594    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091b9ee1-723f-44d7-bac5-629178dbd234-istiod-ca-cert" pod "091b9ee1-723f-44d7-bac5-629178dbd234" (UID: "091b9ee1-723f-44d7-bac5-629178dbd234"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:48.323299 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.322909    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-workload-socket" pod "091b9ee1-723f-44d7-bac5-629178dbd234" (UID: "091b9ee1-723f-44d7-bac5-629178dbd234"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:07:48.323299 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.323232    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:48.323299 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.323248    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:48.323299 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.323258    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:48.323299 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.323269    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:48.323299 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.323280    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091b9ee1-723f-44d7-bac5-629178dbd234-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:48.324901 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.324831    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/091b9ee1-723f-44d7-bac5-629178dbd234-istio-podinfo" pod "091b9ee1-723f-44d7-bac5-629178dbd234" (UID: "091b9ee1-723f-44d7-bac5-629178dbd234"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:07:48.325010 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.324839    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-istio-envoy" pod "091b9ee1-723f-44d7-bac5-629178dbd234" (UID: "091b9ee1-723f-44d7-bac5-629178dbd234"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:07:48.325288 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.325262    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091b9ee1-723f-44d7-bac5-629178dbd234-kube-api-access-fd8dv" pod "091b9ee1-723f-44d7-bac5-629178dbd234" (UID: "091b9ee1-723f-44d7-bac5-629178dbd234"). InnerVolumeSpecName "kube-api-access-fd8dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:48.325674 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.325656    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091b9ee1-723f-44d7-bac5-629178dbd234-istio-token" pod "091b9ee1-723f-44d7-bac5-629178dbd234" (UID: "091b9ee1-723f-44d7-bac5-629178dbd234"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:48.360430 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.360401    4823 generic.go:356] "Generic (PLEG): container finished" podID="091b9ee1-723f-44d7-bac5-629178dbd234" containerID="a8db6bb9b415b6b60261828b6ac44c80194095b9c245aed2712caf1acee58521" exitCode=0
Oct 30 20:07:48.360569 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.360455    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"
Oct 30 20:07:48.360569 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.360498    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws" event={"ID":"091b9ee1-723f-44d7-bac5-629178dbd234","Type":"ContainerDied","Data":"a8db6bb9b415b6b60261828b6ac44c80194095b9c245aed2712caf1acee58521"}
Oct 30 20:07:48.360697 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.360569    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws" event={"ID":"091b9ee1-723f-44d7-bac5-629178dbd234","Type":"ContainerDied","Data":"080255860f3e39bf5aef44b90ca1987ece5cc6558ed1e6737007b1294cb28715"}
Oct 30 20:07:48.360697 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.360600    4823 scope.go:122] "RemoveContainer" containerID="a8db6bb9b415b6b60261828b6ac44c80194095b9c245aed2712caf1acee58521"
Oct 30 20:07:48.377028 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.377005    4823 scope.go:122] "RemoveContainer" containerID="16c71fad5a5ad82b8bf869736e059b942aff2723a3249185755840d3110a9b79"
Oct 30 20:07:48.389021 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.389001    4823 scope.go:122] "RemoveContainer" containerID="a8db6bb9b415b6b60261828b6ac44c80194095b9c245aed2712caf1acee58521"
Oct 30 20:07:48.389363 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:07:48.389343    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8db6bb9b415b6b60261828b6ac44c80194095b9c245aed2712caf1acee58521\": container with ID starting with a8db6bb9b415b6b60261828b6ac44c80194095b9c245aed2712caf1acee58521 not found: ID does not exist" containerID="a8db6bb9b415b6b60261828b6ac44c80194095b9c245aed2712caf1acee58521"
Oct 30 20:07:48.389426 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.389371    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8db6bb9b415b6b60261828b6ac44c80194095b9c245aed2712caf1acee58521"} err="failed to get container status \"a8db6bb9b415b6b60261828b6ac44c80194095b9c245aed2712caf1acee58521\": rpc error: code = NotFound desc = could not find container \"a8db6bb9b415b6b60261828b6ac44c80194095b9c245aed2712caf1acee58521\": container with ID starting with a8db6bb9b415b6b60261828b6ac44c80194095b9c245aed2712caf1acee58521 not found: ID does not exist"
Oct 30 20:07:48.389426 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.389387    4823 scope.go:122] "RemoveContainer" containerID="16c71fad5a5ad82b8bf869736e059b942aff2723a3249185755840d3110a9b79"
Oct 30 20:07:48.389678 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:07:48.389662    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16c71fad5a5ad82b8bf869736e059b942aff2723a3249185755840d3110a9b79\": container with ID starting with 16c71fad5a5ad82b8bf869736e059b942aff2723a3249185755840d3110a9b79 not found: ID does not exist" containerID="16c71fad5a5ad82b8bf869736e059b942aff2723a3249185755840d3110a9b79"
Oct 30 20:07:48.389744 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.389682    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16c71fad5a5ad82b8bf869736e059b942aff2723a3249185755840d3110a9b79"} err="failed to get container status \"16c71fad5a5ad82b8bf869736e059b942aff2723a3249185755840d3110a9b79\": rpc error: code = NotFound desc = could not find container \"16c71fad5a5ad82b8bf869736e059b942aff2723a3249185755840d3110a9b79\": container with ID starting with 16c71fad5a5ad82b8bf869736e059b942aff2723a3249185755840d3110a9b79 not found: ID does not exist"
Oct 30 20:07:48.394981 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.394944    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"]
Oct 30 20:07:48.417128 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.417096    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hhbws"]
Oct 30 20:07:48.425188 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.425148    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd8dv\" (UniqueName: \"kubernetes.io/projected/091b9ee1-723f-44d7-bac5-629178dbd234-kube-api-access-fd8dv\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:48.425188 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.425178    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091b9ee1-723f-44d7-bac5-629178dbd234-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:48.425188 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.425193    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091b9ee1-723f-44d7-bac5-629178dbd234-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:48.425408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:48.425207    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091b9ee1-723f-44d7-bac5-629178dbd234-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:50.281010 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:50.280973    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="091b9ee1-723f-44d7-bac5-629178dbd234" path="/var/lib/kubelet/pods/091b9ee1-723f-44d7-bac5-629178dbd234/volumes"
Oct 30 20:07:53.701270 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:53.701217    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:53.703208 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:53.703173    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:53.815326 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:53.815283    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:53.828840 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:53.827951    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 20:07:54.074061 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:07:54.073960    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:02.016343 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:08:02.016307    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:08:02.427197 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:02.427163    4823 scope.go:122] "RemoveContainer" containerID="0d7854199d14b38c4a13b310ac64460c57099c1ec2b413754ca5dacbb4964e4a"
Oct 30 20:08:02.441918 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:02.441894    4823 scope.go:122] "RemoveContainer" containerID="3e5cc4a0732bbaccf43aea27ebfe935878593c2eb7205f1b3c0f306d6d14b5e6"
Oct 30 20:08:02.452377 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:02.452346    4823 scope.go:122] "RemoveContainer" containerID="ebd738347a61c2eade6e612b012d8de11364f4b82baace8757a4fdf0471a2943"
Oct 30 20:08:02.461781 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:02.461764    4823 scope.go:122] "RemoveContainer" containerID="88a31d81ec2fd051a7308e794d1792039203c7e4d32becc4513d7fbb31542899"
Oct 30 20:08:02.470924 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:02.470856    4823 scope.go:122] "RemoveContainer" containerID="35070bc8e31926d17d8fa3a91bad19c1aa3b690d9920db8ce69e0076af573032"
Oct 30 20:08:02.480416 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:02.480394    4823 scope.go:122] "RemoveContainer" containerID="522140d6bce10b9f4ff4110a959e67881f70a0979a0667457a2fa7168a6b112d"
Oct 30 20:08:09.360447 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.360413    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"]
Oct 30 20:08:09.361257 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.361240    4823 cpu_manager.go:414] "containerMap: removing container" podUID="091b9ee1-723f-44d7-bac5-629178dbd234" containerName="istio-proxy"
Oct 30 20:08:09.361257 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.361257    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091b9ee1-723f-44d7-bac5-629178dbd234" containerName="istio-proxy"
Oct 30 20:08:09.361366 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.361284    4823 cpu_manager.go:414] "containerMap: removing container" podUID="091b9ee1-723f-44d7-bac5-629178dbd234" containerName="istio-proxy"
Oct 30 20:08:09.361366 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.361289    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091b9ee1-723f-44d7-bac5-629178dbd234" containerName="istio-proxy"
Oct 30 20:08:09.361442 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.361433    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="091b9ee1-723f-44d7-bac5-629178dbd234" containerName="istio-proxy"
Oct 30 20:08:09.361482 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.361447    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="091b9ee1-723f-44d7-bac5-629178dbd234" containerName="istio-proxy"
Oct 30 20:08:09.367022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.367004    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.373751 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.373727    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6986a4-openshift-default-dockercfg-rwt7k\""
Oct 30 20:08:09.376125 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.376105    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"]
Oct 30 20:08:09.418052 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.418017    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-workload-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.418261 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.418093    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-data\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.418261 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.418161    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb278\" (UniqueName: \"kubernetes.io/projected/98c1ea21-d79b-457c-95f7-45fd2b2befd7-kube-api-access-zb278\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.418388 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.418313    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-credential-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.418435 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.418414    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-token\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.418497 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.418483    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-podinfo\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.418543 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.418518    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-workload-certs\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.418674 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.418651    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istiod-ca-cert\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.418729 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.418712    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-envoy\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.519807 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.519770    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-data\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.519807 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.519809    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb278\" (UniqueName: \"kubernetes.io/projected/98c1ea21-d79b-457c-95f7-45fd2b2befd7-kube-api-access-zb278\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.520061 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.519901    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-credential-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.520061 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.519967    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-token\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.520191 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.520079    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-podinfo\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.520191 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.520121    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-data\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.520191 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.520133    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-workload-certs\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.520344 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.520197    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istiod-ca-cert\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.520344 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.520251    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-credential-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.520344 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.520252    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-envoy\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.520464 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.520412    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-workload-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.520464 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.520444    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-workload-certs\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.520754 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.520727    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-workload-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.521013 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.520988    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istiod-ca-cert\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.522493 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.522471    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-envoy\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.522720 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.522701    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-podinfo\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.531827 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.531800    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-token\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.531985 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.531966    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb278\" (UniqueName: \"kubernetes.io/projected/98c1ea21-d79b-457c-95f7-45fd2b2befd7-kube-api-access-zb278\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.684343 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.684249    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:09.986646 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.986614    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"]
Oct 30 20:08:09.987079 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:08:09.987055    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c1ea21_d79b_457c_95f7_45fd2b2befd7.slice/crio-04ca6a66b480524881128f4e713996a043782603396b6c7a652a7202b708e28a WatchSource:0}: Error finding container 04ca6a66b480524881128f4e713996a043782603396b6c7a652a7202b708e28a: Status 404 returned error can't find the container with id 04ca6a66b480524881128f4e713996a043782603396b6c7a652a7202b708e28a
Oct 30 20:08:09.990098 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.990064    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:08:09.990197 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.990133    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:08:09.990197 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:09.990173    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:08:10.664589 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:10.664551    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" event={"ID":"98c1ea21-d79b-457c-95f7-45fd2b2befd7","Type":"ContainerStarted","Data":"85488a16a7ebe75413316d628210bcd7c66c7c19885aa408a882c9732f4f3740"}
Oct 30 20:08:10.665076 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:10.664605    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" event={"ID":"98c1ea21-d79b-457c-95f7-45fd2b2befd7","Type":"ContainerStarted","Data":"04ca6a66b480524881128f4e713996a043782603396b6c7a652a7202b708e28a"}
Oct 30 20:08:10.684532 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:10.684504    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:10.689314 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:10.689257    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podStartSLOduration=1.689239802 podStartE2EDuration="1.689239802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:10.686846293 +0000 UTC m=+2290.579460637" watchObservedRunningTime="2026-10-30 20:08:10.689239802 +0000 UTC m=+2290.581854130"
Oct 30 20:08:11.686854 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:11.686821    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:11.687340 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:11.686910    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:12.685638 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:12.685604    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:12.685842 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:12.685660    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:13.684733 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:13.684695    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:13.685256 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:13.684764    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:14.684950 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:14.684909    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:14.685431 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:14.684968    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:15.685478 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:15.685443    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:15.685912 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:15.685496    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:16.684839 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:16.684797    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:16.685111 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:16.684890    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:17.685704 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:17.685667    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:17.686142 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:17.685732    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:18.685216 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:18.685172    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:18.685404 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:18.685228    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:19.685461 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:19.685424    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:19.685979 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:19.685512    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:20.685171 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:20.685137    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:20.685351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:20.685198    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:21.685618 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:21.685584    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:21.686131 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:21.685648    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:22.685218 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:22.685182    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:22.685411 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:22.685239    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:23.685630 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:23.685598    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:23.686112 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:23.685647    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:24.686143 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:24.686098    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:24.686700 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:24.686365    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:24.686952 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:24.686923    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:25.692990 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:25.692949    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:25.693641 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:25.693608    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:26.685313 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:26.685271    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:26.685504 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:26.685361    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:27.684680 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:27.684644    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:27.685127 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:27.684696    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:27.954690 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:27.954610    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ntzq9"]
Oct 30 20:08:27.954848 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:27.954830    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/odh-model-controller-769875db56-ntzq9" podUID="aec94275-86c6-43e2-88d5-ceeed0756ebd" containerName="manager" containerID="cri-o://0d8f691871d3ba8c0eef0813249391f2a968ab8a3393c6e3f4f3985fcf5b3938" gracePeriod=10
Oct 30 20:08:28.273227 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.273201    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ntzq9"
Oct 30 20:08:28.376697 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.376661    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aec94275-86c6-43e2-88d5-ceeed0756ebd-cert\" (UniqueName: \"kubernetes.io/secret/aec94275-86c6-43e2-88d5-ceeed0756ebd-cert\") pod \"aec94275-86c6-43e2-88d5-ceeed0756ebd\" (UID: \"aec94275-86c6-43e2-88d5-ceeed0756ebd\") "
Oct 30 20:08:28.376877 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.376787    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aec94275-86c6-43e2-88d5-ceeed0756ebd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aec94275-86c6-43e2-88d5-ceeed0756ebd-metrics-certs\") pod \"aec94275-86c6-43e2-88d5-ceeed0756ebd\" (UID: \"aec94275-86c6-43e2-88d5-ceeed0756ebd\") "
Oct 30 20:08:28.376968 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.376907    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec94275-86c6-43e2-88d5-ceeed0756ebd-kube-api-access-f9p9z\" (UniqueName: \"kubernetes.io/projected/aec94275-86c6-43e2-88d5-ceeed0756ebd-kube-api-access-f9p9z\") pod \"aec94275-86c6-43e2-88d5-ceeed0756ebd\" (UID: \"aec94275-86c6-43e2-88d5-ceeed0756ebd\") "
Oct 30 20:08:28.379777 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.379744    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec94275-86c6-43e2-88d5-ceeed0756ebd-kube-api-access-f9p9z" pod "aec94275-86c6-43e2-88d5-ceeed0756ebd" (UID: "aec94275-86c6-43e2-88d5-ceeed0756ebd"). InnerVolumeSpecName "kube-api-access-f9p9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:28.379945 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.379823    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec94275-86c6-43e2-88d5-ceeed0756ebd-cert" pod "aec94275-86c6-43e2-88d5-ceeed0756ebd" (UID: "aec94275-86c6-43e2-88d5-ceeed0756ebd"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:28.380233 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.380210    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec94275-86c6-43e2-88d5-ceeed0756ebd-metrics-certs" pod "aec94275-86c6-43e2-88d5-ceeed0756ebd" (UID: "aec94275-86c6-43e2-88d5-ceeed0756ebd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:28.480001 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.479959    4823 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aec94275-86c6-43e2-88d5-ceeed0756ebd-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.480001 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.479991    4823 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aec94275-86c6-43e2-88d5-ceeed0756ebd-metrics-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.480001 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.480007    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9p9z\" (UniqueName: \"kubernetes.io/projected/aec94275-86c6-43e2-88d5-ceeed0756ebd-kube-api-access-f9p9z\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:28.685114 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.685080    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:28.685499 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.685136    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:28.823698 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.823652    4823 generic.go:356] "Generic (PLEG): container finished" podID="aec94275-86c6-43e2-88d5-ceeed0756ebd" containerID="0d8f691871d3ba8c0eef0813249391f2a968ab8a3393c6e3f4f3985fcf5b3938" exitCode=0
Oct 30 20:08:28.823905 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.823717    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ntzq9"
Oct 30 20:08:28.823905 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.823729    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ntzq9" event={"ID":"aec94275-86c6-43e2-88d5-ceeed0756ebd","Type":"ContainerDied","Data":"0d8f691871d3ba8c0eef0813249391f2a968ab8a3393c6e3f4f3985fcf5b3938"}
Oct 30 20:08:28.823905 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.823817    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ntzq9" event={"ID":"aec94275-86c6-43e2-88d5-ceeed0756ebd","Type":"ContainerDied","Data":"57058f45368a12c676b28b818a8b003b574b219364621366e99bcb10a683c719"}
Oct 30 20:08:28.823905 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.823842    4823 scope.go:122] "RemoveContainer" containerID="0d8f691871d3ba8c0eef0813249391f2a968ab8a3393c6e3f4f3985fcf5b3938"
Oct 30 20:08:28.847504 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.847480    4823 scope.go:122] "RemoveContainer" containerID="0d8f691871d3ba8c0eef0813249391f2a968ab8a3393c6e3f4f3985fcf5b3938"
Oct 30 20:08:28.847805 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:08:28.847776    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8f691871d3ba8c0eef0813249391f2a968ab8a3393c6e3f4f3985fcf5b3938\": container with ID starting with 0d8f691871d3ba8c0eef0813249391f2a968ab8a3393c6e3f4f3985fcf5b3938 not found: ID does not exist" containerID="0d8f691871d3ba8c0eef0813249391f2a968ab8a3393c6e3f4f3985fcf5b3938"
Oct 30 20:08:28.847853 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.847821    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8f691871d3ba8c0eef0813249391f2a968ab8a3393c6e3f4f3985fcf5b3938"} err="failed to get container status \"0d8f691871d3ba8c0eef0813249391f2a968ab8a3393c6e3f4f3985fcf5b3938\": rpc error: code = NotFound desc = could not find container \"0d8f691871d3ba8c0eef0813249391f2a968ab8a3393c6e3f4f3985fcf5b3938\": container with ID starting with 0d8f691871d3ba8c0eef0813249391f2a968ab8a3393c6e3f4f3985fcf5b3938 not found: ID does not exist"
Oct 30 20:08:28.855221 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.855198    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ntzq9"]
Oct 30 20:08:28.861765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:28.861742    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ntzq9"]
Oct 30 20:08:29.202534 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.202497    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-dxfcj"]
Oct 30 20:08:29.203459 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.203437    4823 cpu_manager.go:414] "containerMap: removing container" podUID="aec94275-86c6-43e2-88d5-ceeed0756ebd" containerName="manager"
Oct 30 20:08:29.203532 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.203461    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec94275-86c6-43e2-88d5-ceeed0756ebd" containerName="manager"
Oct 30 20:08:29.203635 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.203625    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec94275-86c6-43e2-88d5-ceeed0756ebd" containerName="manager"
Oct 30 20:08:29.209204 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.209176    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-dxfcj"
Oct 30 20:08:29.216525 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.216497    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-dxfcj"]
Oct 30 20:08:29.216675 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.216591    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 20:08:29.219226 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.219205    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bkdcs\""
Oct 30 20:08:29.293947 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.293902    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d945d1f6-e8e1-45fb-94e2-fe7d2751a813-cert\") pod \"odh-model-controller-769875db56-dxfcj\" (UID: \"d945d1f6-e8e1-45fb-94e2-fe7d2751a813\") " pod="opendatahub/odh-model-controller-769875db56-dxfcj"
Oct 30 20:08:29.294128 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.294037    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d945d1f6-e8e1-45fb-94e2-fe7d2751a813-metrics-certs\") pod \"odh-model-controller-769875db56-dxfcj\" (UID: \"d945d1f6-e8e1-45fb-94e2-fe7d2751a813\") " pod="opendatahub/odh-model-controller-769875db56-dxfcj"
Oct 30 20:08:29.294190 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.294137    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqk9g\" (UniqueName: \"kubernetes.io/projected/d945d1f6-e8e1-45fb-94e2-fe7d2751a813-kube-api-access-wqk9g\") pod \"odh-model-controller-769875db56-dxfcj\" (UID: \"d945d1f6-e8e1-45fb-94e2-fe7d2751a813\") " pod="opendatahub/odh-model-controller-769875db56-dxfcj"
Oct 30 20:08:29.317333 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.317308    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 20:08:29.395719 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.395677    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d945d1f6-e8e1-45fb-94e2-fe7d2751a813-cert\") pod \"odh-model-controller-769875db56-dxfcj\" (UID: \"d945d1f6-e8e1-45fb-94e2-fe7d2751a813\") " pod="opendatahub/odh-model-controller-769875db56-dxfcj"
Oct 30 20:08:29.395956 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.395742    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d945d1f6-e8e1-45fb-94e2-fe7d2751a813-metrics-certs\") pod \"odh-model-controller-769875db56-dxfcj\" (UID: \"d945d1f6-e8e1-45fb-94e2-fe7d2751a813\") " pod="opendatahub/odh-model-controller-769875db56-dxfcj"
Oct 30 20:08:29.395956 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.395802    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqk9g\" (UniqueName: \"kubernetes.io/projected/d945d1f6-e8e1-45fb-94e2-fe7d2751a813-kube-api-access-wqk9g\") pod \"odh-model-controller-769875db56-dxfcj\" (UID: \"d945d1f6-e8e1-45fb-94e2-fe7d2751a813\") " pod="opendatahub/odh-model-controller-769875db56-dxfcj"
Oct 30 20:08:29.398381 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.398352    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d945d1f6-e8e1-45fb-94e2-fe7d2751a813-metrics-certs\") pod \"odh-model-controller-769875db56-dxfcj\" (UID: \"d945d1f6-e8e1-45fb-94e2-fe7d2751a813\") " pod="opendatahub/odh-model-controller-769875db56-dxfcj"
Oct 30 20:08:29.398488 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.398403    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d945d1f6-e8e1-45fb-94e2-fe7d2751a813-cert\") pod \"odh-model-controller-769875db56-dxfcj\" (UID: \"d945d1f6-e8e1-45fb-94e2-fe7d2751a813\") " pod="opendatahub/odh-model-controller-769875db56-dxfcj"
Oct 30 20:08:29.408016 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.407996    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqk9g\" (UniqueName: \"kubernetes.io/projected/d945d1f6-e8e1-45fb-94e2-fe7d2751a813-kube-api-access-wqk9g\") pod \"odh-model-controller-769875db56-dxfcj\" (UID: \"d945d1f6-e8e1-45fb-94e2-fe7d2751a813\") " pod="opendatahub/odh-model-controller-769875db56-dxfcj"
Oct 30 20:08:29.534758 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.534666    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-dxfcj"
Oct 30 20:08:29.685156 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.685117    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:29.685541 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.685176    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:29.828594 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:29.828562    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-dxfcj"]
Oct 30 20:08:29.829405 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:08:29.829377    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd945d1f6_e8e1_45fb_94e2_fe7d2751a813.slice/crio-4bb7feb2e05f94f8c67f194053468bee4fd5e654760e029b7d745e86e04ce4b5 WatchSource:0}: Error finding container 4bb7feb2e05f94f8c67f194053468bee4fd5e654760e029b7d745e86e04ce4b5: Status 404 returned error can't find the container with id 4bb7feb2e05f94f8c67f194053468bee4fd5e654760e029b7d745e86e04ce4b5
Oct 30 20:08:30.285750 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:30.282946    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec94275-86c6-43e2-88d5-ceeed0756ebd" path="/var/lib/kubelet/pods/aec94275-86c6-43e2-88d5-ceeed0756ebd/volumes"
Oct 30 20:08:30.685408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:30.685374    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:08:30.685907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:30.685432    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:08:30.847214 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:30.847177    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-dxfcj" event={"ID":"d945d1f6-e8e1-45fb-94e2-fe7d2751a813","Type":"ContainerStarted","Data":"2d0770eb8af34f3198d4e569846efa21a4778e91c35ffc642a4e179ff88d5ff3"}
Oct 30 20:08:30.847395 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:30.847239    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-dxfcj" event={"ID":"d945d1f6-e8e1-45fb-94e2-fe7d2751a813","Type":"ContainerStarted","Data":"4bb7feb2e05f94f8c67f194053468bee4fd5e654760e029b7d745e86e04ce4b5"}
Oct 30 20:08:30.847395 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:30.847268    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-dxfcj"
Oct 30 20:08:30.871022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:30.870973    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-dxfcj" podStartSLOduration=1.459263184 podStartE2EDuration="1.870958501s" totalImagesPullingTime="411.695317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:30.865120097 +0000 UTC m=+2310.757734411" watchObservedRunningTime="2026-10-30 20:08:30.870958501 +0000 UTC m=+2310.763572842"
Oct 30 20:08:31.690357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:31.690329    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:31.691654 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:31.691632    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:08:32.016629 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:08:32.016545    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:08:35.034975 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:35.034933    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"]
Oct 30 20:08:35.039415 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:35.039391    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660ac090-9ssrt"]
Oct 30 20:08:36.281659 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:36.281626    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08bc865f-a124-4e5d-9cdc-a296aac31513" path="/var/lib/kubelet/pods/08bc865f-a124-4e5d-9cdc-a296aac31513/volumes"
Oct 30 20:08:41.872683 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:08:41.872647    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-dxfcj"
Oct 30 20:09:02.020311 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:09:02.020269    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:09:02.540160 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:09:02.540129    4823 scope.go:122] "RemoveContainer" containerID="30429f5299a135d28a54331885cbd1447337b09a218af9c4ede8a6ce09c7d2c8"
Oct 30 20:09:02.551320 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:09:02.551300    4823 scope.go:122] "RemoveContainer" containerID="8a6e7b3bafa9fd04d4a9503185f1196149094ca84348f93750bd5c9a7e7859d0"
Oct 30 20:09:32.016762 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:09:32.016724    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:10:01.468008 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:01.467971    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:10:01.472407 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:01.472377    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:10:02.080227 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:10:02.076051    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:10:02.590102 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:02.590071    4823 scope.go:122] "RemoveContainer" containerID="204ae4e2c4de9294ea9160ad2711c9f4a70610f14c449a6392ce6f5124b43769"
Oct 30 20:10:02.601161 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:02.601138    4823 scope.go:122] "RemoveContainer" containerID="3a7e3009d2ffcb12d308c407f743e5fb005fc2305471e10d10cbb285bbe44d92"
Oct 30 20:10:07.932716 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:07.932682    4823 ???:1] "http: TLS handshake error from 10.0.139.103:46464: read tcp 10.0.138.61:10250->10.0.139.103:46464: read: connection reset by peer"
Oct 30 20:10:08.689190 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:08.689160    4823 ???:1] "http: TLS handshake error from 10.0.139.103:46271: EOF"
Oct 30 20:10:09.247986 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:09.247955    4823 ???:1] "http: TLS handshake error from 10.0.139.103:52042: EOF"
Oct 30 20:10:16.664388 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:16.662983    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"]
Oct 30 20:10:16.664388 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:16.663371    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv" podUID="3bc11d6a-c12d-4dab-8356-a8a05ee82f37" containerName="istio-proxy" containerID="cri-o://2e37ea1ce0eeeb34712b575c3728d81eb5656a9a36f6a3e8b9c5d387e45779cc" gracePeriod=30
Oct 30 20:10:16.678728 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:16.678696    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"]
Oct 30 20:10:16.679203 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:16.679173    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p" podUID="6df66169-610e-4ff9-8706-25e02d655b48" containerName="istio-proxy" containerID="cri-o://9cd9b1c5acdf48457bc4f6b5f47b8f9d7953971c9aad05516ef2e2a1720a46b0" gracePeriod=30
Oct 30 20:10:16.757335 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:16.757297    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"]
Oct 30 20:10:16.757597 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:16.757548    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg" podUID="11656a86-294a-4cd1-a29c-6f8d02a39c2b" containerName="istio-proxy" containerID="cri-o://9fcfd6ff1c9e699ceba6fe0f726cd6e4da9785667e55519cd9e8291e16dca3d6" gracePeriod=30
Oct 30 20:10:16.911842 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:16.911801    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"]
Oct 30 20:10:16.912081 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:16.912048    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd" podUID="be418f6d-e900-479d-bbab-3b94ffba7a7f" containerName="discovery" containerID="cri-o://811ab074e2209ec412937a682fb3cef17549565a25e0ed3ba1c6f13a16f5d276" gracePeriod=30
Oct 30 20:10:17.285426 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.285398    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 20:10:17.312745 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.312710    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-token\" (UniqueName: \"kubernetes.io/projected/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-token\") pod \"be418f6d-e900-479d-bbab-3b94ffba7a7f\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") "
Oct 30 20:10:17.312940 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.312789    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be418f6d-e900-479d-bbab-3b94ffba7a7f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be418f6d-e900-479d-bbab-3b94ffba7a7f-local-certs\") pod \"be418f6d-e900-479d-bbab-3b94ffba7a7f\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") "
Oct 30 20:10:17.312940 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.312884    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-csr-ca-configmap\") pod \"be418f6d-e900-479d-bbab-3b94ffba7a7f\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") "
Oct 30 20:10:17.313086 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.313058    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-cacerts\" (UniqueName: \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-cacerts\") pod \"be418f6d-e900-479d-bbab-3b94ffba7a7f\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") "
Oct 30 20:10:17.313149 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.313090    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be418f6d-e900-479d-bbab-3b94ffba7a7f-kube-api-access-4n7jr\" (UniqueName: \"kubernetes.io/projected/be418f6d-e900-479d-bbab-3b94ffba7a7f-kube-api-access-4n7jr\") pod \"be418f6d-e900-479d-bbab-3b94ffba7a7f\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") "
Oct 30 20:10:17.313243 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.313195    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-kubeconfig\") pod \"be418f6d-e900-479d-bbab-3b94ffba7a7f\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") "
Oct 30 20:10:17.313323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.313277    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-csr-ca-configmap" pod "be418f6d-e900-479d-bbab-3b94ffba7a7f" (UID: "be418f6d-e900-479d-bbab-3b94ffba7a7f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:17.313380 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.313365    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-csr-dns-cert\") pod \"be418f6d-e900-479d-bbab-3b94ffba7a7f\" (UID: \"be418f6d-e900-479d-bbab-3b94ffba7a7f\") "
Oct 30 20:10:17.314317 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.314288    4823 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-csr-ca-configmap\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:17.315671 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.315643    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be418f6d-e900-479d-bbab-3b94ffba7a7f-local-certs" pod "be418f6d-e900-479d-bbab-3b94ffba7a7f" (UID: "be418f6d-e900-479d-bbab-3b94ffba7a7f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:17.315779 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.315732    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-token" pod "be418f6d-e900-479d-bbab-3b94ffba7a7f" (UID: "be418f6d-e900-479d-bbab-3b94ffba7a7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:17.315927 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.315846    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be418f6d-e900-479d-bbab-3b94ffba7a7f-kube-api-access-4n7jr" pod "be418f6d-e900-479d-bbab-3b94ffba7a7f" (UID: "be418f6d-e900-479d-bbab-3b94ffba7a7f"). InnerVolumeSpecName "kube-api-access-4n7jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:17.316178 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.316135    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-cacerts" pod "be418f6d-e900-479d-bbab-3b94ffba7a7f" (UID: "be418f6d-e900-479d-bbab-3b94ffba7a7f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:17.316178 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.316160    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-kubeconfig" pod "be418f6d-e900-479d-bbab-3b94ffba7a7f" (UID: "be418f6d-e900-479d-bbab-3b94ffba7a7f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:17.317260 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.317229    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-csr-dns-cert" pod "be418f6d-e900-479d-bbab-3b94ffba7a7f" (UID: "be418f6d-e900-479d-bbab-3b94ffba7a7f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:17.416284 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.416243    4823 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-cacerts\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:17.416284 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.416277    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n7jr\" (UniqueName: \"kubernetes.io/projected/be418f6d-e900-479d-bbab-3b94ffba7a7f-kube-api-access-4n7jr\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:17.416284 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.416291    4823 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-kubeconfig\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:17.416549 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.416301    4823 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-csr-dns-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:17.416549 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.416310    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be418f6d-e900-479d-bbab-3b94ffba7a7f-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:17.416549 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:17.416318    4823 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be418f6d-e900-479d-bbab-3b94ffba7a7f-local-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:18.029856 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:18.029819    4823 generic.go:356] "Generic (PLEG): container finished" podID="be418f6d-e900-479d-bbab-3b94ffba7a7f" containerID="811ab074e2209ec412937a682fb3cef17549565a25e0ed3ba1c6f13a16f5d276" exitCode=0
Oct 30 20:10:18.029856 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:18.029875    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"
Oct 30 20:10:18.030365 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:18.029898    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd" event={"ID":"be418f6d-e900-479d-bbab-3b94ffba7a7f","Type":"ContainerDied","Data":"811ab074e2209ec412937a682fb3cef17549565a25e0ed3ba1c6f13a16f5d276"}
Oct 30 20:10:18.030365 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:18.029976    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd" event={"ID":"be418f6d-e900-479d-bbab-3b94ffba7a7f","Type":"ContainerDied","Data":"36833c96bd78fe69de9825b8831c346525dafe2c98e86bce573cb5d3b604061f"}
Oct 30 20:10:18.030365 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:18.030004    4823 scope.go:122] "RemoveContainer" containerID="811ab074e2209ec412937a682fb3cef17549565a25e0ed3ba1c6f13a16f5d276"
Oct 30 20:10:18.042979 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:18.042958    4823 scope.go:122] "RemoveContainer" containerID="811ab074e2209ec412937a682fb3cef17549565a25e0ed3ba1c6f13a16f5d276"
Oct 30 20:10:18.043272 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:10:18.043253    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"811ab074e2209ec412937a682fb3cef17549565a25e0ed3ba1c6f13a16f5d276\": container with ID starting with 811ab074e2209ec412937a682fb3cef17549565a25e0ed3ba1c6f13a16f5d276 not found: ID does not exist" containerID="811ab074e2209ec412937a682fb3cef17549565a25e0ed3ba1c6f13a16f5d276"
Oct 30 20:10:18.043330 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:18.043283    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"811ab074e2209ec412937a682fb3cef17549565a25e0ed3ba1c6f13a16f5d276"} err="failed to get container status \"811ab074e2209ec412937a682fb3cef17549565a25e0ed3ba1c6f13a16f5d276\": rpc error: code = NotFound desc = could not find container \"811ab074e2209ec412937a682fb3cef17549565a25e0ed3ba1c6f13a16f5d276\": container with ID starting with 811ab074e2209ec412937a682fb3cef17549565a25e0ed3ba1c6f13a16f5d276 not found: ID does not exist"
Oct 30 20:10:18.045911 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:18.045887    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:10:18.046008 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:18.045953    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p" podUID="6df66169-610e-4ff9-8706-25e02d655b48" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:10:18.064758 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:18.064734    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"]
Oct 30 20:10:18.074005 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:18.073980    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b26sd"]
Oct 30 20:10:18.281080 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:18.281002    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be418f6d-e900-479d-bbab-3b94ffba7a7f" path="/var/lib/kubelet/pods/be418f6d-e900-479d-bbab-3b94ffba7a7f/volumes"
Oct 30 20:10:18.438064 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:18.438028    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-4zvvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:10:18.438266 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:18.438090    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg" podUID="11656a86-294a-4cd1-a29c-6f8d02a39c2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:10:18.974400 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:18.974363    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:10:18.974576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:18.974439    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv" podUID="3bc11d6a-c12d-4dab-8356-a8a05ee82f37" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:10:21.997442 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:21.997086    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 20:10:22.088716 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.088683    4823 generic.go:356] "Generic (PLEG): container finished" podID="11656a86-294a-4cd1-a29c-6f8d02a39c2b" containerID="9fcfd6ff1c9e699ceba6fe0f726cd6e4da9785667e55519cd9e8291e16dca3d6" exitCode=0
Oct 30 20:10:22.088940 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.088759    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg" event={"ID":"11656a86-294a-4cd1-a29c-6f8d02a39c2b","Type":"ContainerDied","Data":"9fcfd6ff1c9e699ceba6fe0f726cd6e4da9785667e55519cd9e8291e16dca3d6"}
Oct 30 20:10:22.091078 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.090643    4823 generic.go:356] "Generic (PLEG): container finished" podID="3bc11d6a-c12d-4dab-8356-a8a05ee82f37" containerID="2e37ea1ce0eeeb34712b575c3728d81eb5656a9a36f6a3e8b9c5d387e45779cc" exitCode=0
Oct 30 20:10:22.091078 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.090780    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv" event={"ID":"3bc11d6a-c12d-4dab-8356-a8a05ee82f37","Type":"ContainerDied","Data":"2e37ea1ce0eeeb34712b575c3728d81eb5656a9a36f6a3e8b9c5d387e45779cc"}
Oct 30 20:10:22.091078 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.090819    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv" event={"ID":"3bc11d6a-c12d-4dab-8356-a8a05ee82f37","Type":"ContainerDied","Data":"d21a7525f2346526b3a189a9079cabc06a5de91b0f8637aec0cf4ac0677f79da"}
Oct 30 20:10:22.091078 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.090842    4823 scope.go:122] "RemoveContainer" containerID="2e37ea1ce0eeeb34712b575c3728d81eb5656a9a36f6a3e8b9c5d387e45779cc"
Oct 30 20:10:22.091078 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.091059    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"
Oct 30 20:10:22.092900 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.092791    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-data\") pod \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") "
Oct 30 20:10:22.093033 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.092906    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-kube-api-access-rd5z4\" (UniqueName: \"kubernetes.io/projected/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-kube-api-access-rd5z4\") pod \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") "
Oct 30 20:10:22.093033 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.093011    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-credential-socket\") pod \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") "
Oct 30 20:10:22.093200 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.093058    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-workload-socket\") pod \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") "
Oct 30 20:10:22.093261 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.093208    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istiod-ca-cert\") pod \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") "
Oct 30 20:10:22.093319 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.093264    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-envoy\") pod \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") "
Oct 30 20:10:22.093387 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.093364    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-token\" (UniqueName: \"kubernetes.io/projected/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-token\") pod \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") "
Oct 30 20:10:22.093558 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.093453    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-workload-certs\") pod \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") "
Oct 30 20:10:22.093674 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.093588    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-podinfo\") pod \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\" (UID: \"3bc11d6a-c12d-4dab-8356-a8a05ee82f37\") "
Oct 30 20:10:22.093769 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.093714    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-credential-socket" pod "3bc11d6a-c12d-4dab-8356-a8a05ee82f37" (UID: "3bc11d6a-c12d-4dab-8356-a8a05ee82f37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.094471 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.093939    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-workload-certs" pod "3bc11d6a-c12d-4dab-8356-a8a05ee82f37" (UID: "3bc11d6a-c12d-4dab-8356-a8a05ee82f37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.094471 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.094244    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istiod-ca-cert" pod "3bc11d6a-c12d-4dab-8356-a8a05ee82f37" (UID: "3bc11d6a-c12d-4dab-8356-a8a05ee82f37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:22.094471 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.094261    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-workload-socket" pod "3bc11d6a-c12d-4dab-8356-a8a05ee82f37" (UID: "3bc11d6a-c12d-4dab-8356-a8a05ee82f37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.095027 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.095004    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-data" pod "3bc11d6a-c12d-4dab-8356-a8a05ee82f37" (UID: "3bc11d6a-c12d-4dab-8356-a8a05ee82f37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.096656 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.096628    4823 generic.go:356] "Generic (PLEG): container finished" podID="6df66169-610e-4ff9-8706-25e02d655b48" containerID="9cd9b1c5acdf48457bc4f6b5f47b8f9d7953971c9aad05516ef2e2a1720a46b0" exitCode=0
Oct 30 20:10:22.096795 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.096739    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p" event={"ID":"6df66169-610e-4ff9-8706-25e02d655b48","Type":"ContainerDied","Data":"9cd9b1c5acdf48457bc4f6b5f47b8f9d7953971c9aad05516ef2e2a1720a46b0"}
Oct 30 20:10:22.098708 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.098683    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.098819 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.098763    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.098819 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.098784    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.098819 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.098801    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.098819 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.098816    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.100949 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.100787    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-envoy" pod "3bc11d6a-c12d-4dab-8356-a8a05ee82f37" (UID: "3bc11d6a-c12d-4dab-8356-a8a05ee82f37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.100949 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.100787    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-podinfo" pod "3bc11d6a-c12d-4dab-8356-a8a05ee82f37" (UID: "3bc11d6a-c12d-4dab-8356-a8a05ee82f37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:10:22.101072 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.100965    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-token" pod "3bc11d6a-c12d-4dab-8356-a8a05ee82f37" (UID: "3bc11d6a-c12d-4dab-8356-a8a05ee82f37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:22.143107 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.143074    4823 scope.go:122] "RemoveContainer" containerID="2e37ea1ce0eeeb34712b575c3728d81eb5656a9a36f6a3e8b9c5d387e45779cc"
Oct 30 20:10:22.143728 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:10:22.143693    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e37ea1ce0eeeb34712b575c3728d81eb5656a9a36f6a3e8b9c5d387e45779cc\": container with ID starting with 2e37ea1ce0eeeb34712b575c3728d81eb5656a9a36f6a3e8b9c5d387e45779cc not found: ID does not exist" containerID="2e37ea1ce0eeeb34712b575c3728d81eb5656a9a36f6a3e8b9c5d387e45779cc"
Oct 30 20:10:22.143913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.143729    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-kube-api-access-rd5z4" pod "3bc11d6a-c12d-4dab-8356-a8a05ee82f37" (UID: "3bc11d6a-c12d-4dab-8356-a8a05ee82f37"). InnerVolumeSpecName "kube-api-access-rd5z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:22.143913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.143738    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e37ea1ce0eeeb34712b575c3728d81eb5656a9a36f6a3e8b9c5d387e45779cc"} err="failed to get container status \"2e37ea1ce0eeeb34712b575c3728d81eb5656a9a36f6a3e8b9c5d387e45779cc\": rpc error: code = NotFound desc = could not find container \"2e37ea1ce0eeeb34712b575c3728d81eb5656a9a36f6a3e8b9c5d387e45779cc\": container with ID starting with 2e37ea1ce0eeeb34712b575c3728d81eb5656a9a36f6a3e8b9c5d387e45779cc not found: ID does not exist"
Oct 30 20:10:22.160354 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.160317    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 20:10:22.186517 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.186492    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 20:10:22.199907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.199873    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df66169-610e-4ff9-8706-25e02d655b48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df66169-610e-4ff9-8706-25e02d655b48-istiod-ca-cert\") pod \"6df66169-610e-4ff9-8706-25e02d655b48\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") "
Oct 30 20:10:22.200071 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.199919    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-istio-envoy\") pod \"6df66169-610e-4ff9-8706-25e02d655b48\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") "
Oct 30 20:10:22.200071 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.199989    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6df66169-610e-4ff9-8706-25e02d655b48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df66169-610e-4ff9-8706-25e02d655b48-istio-podinfo\") pod \"6df66169-610e-4ff9-8706-25e02d655b48\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") "
Oct 30 20:10:22.200071 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.200038    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-istio-data\") pod \"6df66169-610e-4ff9-8706-25e02d655b48\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") "
Oct 30 20:10:22.200071 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.200069    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df66169-610e-4ff9-8706-25e02d655b48-kube-api-access-4wd96\" (UniqueName: \"kubernetes.io/projected/6df66169-610e-4ff9-8706-25e02d655b48-kube-api-access-4wd96\") pod \"6df66169-610e-4ff9-8706-25e02d655b48\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") "
Oct 30 20:10:22.200316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.200229    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df66169-610e-4ff9-8706-25e02d655b48-istio-token\" (UniqueName: \"kubernetes.io/projected/6df66169-610e-4ff9-8706-25e02d655b48-istio-token\") pod \"6df66169-610e-4ff9-8706-25e02d655b48\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") "
Oct 30 20:10:22.200316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.200272    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-credential-socket\") pod \"6df66169-610e-4ff9-8706-25e02d655b48\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") "
Oct 30 20:10:22.200316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.200278    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df66169-610e-4ff9-8706-25e02d655b48-istiod-ca-cert" pod "6df66169-610e-4ff9-8706-25e02d655b48" (UID: "6df66169-610e-4ff9-8706-25e02d655b48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:22.200553 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.200529    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-istio-data" pod "6df66169-610e-4ff9-8706-25e02d655b48" (UID: "6df66169-610e-4ff9-8706-25e02d655b48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.200627 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.200601    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-workload-certs\") pod \"6df66169-610e-4ff9-8706-25e02d655b48\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") "
Oct 30 20:10:22.200689 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.200651    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-credential-socket" pod "6df66169-610e-4ff9-8706-25e02d655b48" (UID: "6df66169-610e-4ff9-8706-25e02d655b48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.200765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.200746    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-workload-socket\") pod \"6df66169-610e-4ff9-8706-25e02d655b48\" (UID: \"6df66169-610e-4ff9-8706-25e02d655b48\") "
Oct 30 20:10:22.200962 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.200923    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-workload-certs" pod "6df66169-610e-4ff9-8706-25e02d655b48" (UID: "6df66169-610e-4ff9-8706-25e02d655b48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.201191 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.201162    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-workload-socket" pod "6df66169-610e-4ff9-8706-25e02d655b48" (UID: "6df66169-610e-4ff9-8706-25e02d655b48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.201903 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.201845    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df66169-610e-4ff9-8706-25e02d655b48-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.201903 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.201898    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.202052 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.201914    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.202052 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.201931    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd5z4\" (UniqueName: \"kubernetes.io/projected/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-kube-api-access-rd5z4\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.202052 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.201948    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.202052 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.201962    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.202052 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.201979    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.202052 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.201993    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc11d6a-c12d-4dab-8356-a8a05ee82f37-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.202052 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.202007    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.203445 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.203383    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df66169-610e-4ff9-8706-25e02d655b48-istio-token" pod "6df66169-610e-4ff9-8706-25e02d655b48" (UID: "6df66169-610e-4ff9-8706-25e02d655b48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:22.204596 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.204570    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6df66169-610e-4ff9-8706-25e02d655b48-istio-podinfo" pod "6df66169-610e-4ff9-8706-25e02d655b48" (UID: "6df66169-610e-4ff9-8706-25e02d655b48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:10:22.206304 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.206276    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-istio-envoy" pod "6df66169-610e-4ff9-8706-25e02d655b48" (UID: "6df66169-610e-4ff9-8706-25e02d655b48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.207488 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.207465    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df66169-610e-4ff9-8706-25e02d655b48-kube-api-access-4wd96" pod "6df66169-610e-4ff9-8706-25e02d655b48" (UID: "6df66169-610e-4ff9-8706-25e02d655b48"). InnerVolumeSpecName "kube-api-access-4wd96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:22.303192 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.303167    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-envoy\") pod \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") "
Oct 30 20:10:22.303304 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.303263    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-workload-socket\") pod \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") "
Oct 30 20:10:22.303446 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.303430    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11656a86-294a-4cd1-a29c-6f8d02a39c2b-kube-api-access-qgsjj\" (UniqueName: \"kubernetes.io/projected/11656a86-294a-4cd1-a29c-6f8d02a39c2b-kube-api-access-qgsjj\") pod \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") "
Oct 30 20:10:22.303500 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.303473    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-data\") pod \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") "
Oct 30 20:10:22.303557 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.303532    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-workload-socket" pod "11656a86-294a-4cd1-a29c-6f8d02a39c2b" (UID: "11656a86-294a-4cd1-a29c-6f8d02a39c2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.303621 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.303556    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-workload-certs\") pod \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") "
Oct 30 20:10:22.303686 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.303669    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-credential-socket\") pod \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") "
Oct 30 20:10:22.303752 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.303737    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-token\" (UniqueName: \"kubernetes.io/projected/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-token\") pod \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") "
Oct 30 20:10:22.303819 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.303797    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-workload-certs" pod "11656a86-294a-4cd1-a29c-6f8d02a39c2b" (UID: "11656a86-294a-4cd1-a29c-6f8d02a39c2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.303896 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.303816    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-data" pod "11656a86-294a-4cd1-a29c-6f8d02a39c2b" (UID: "11656a86-294a-4cd1-a29c-6f8d02a39c2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.303896 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.303842    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istiod-ca-cert\") pod \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") "
Oct 30 20:10:22.304082 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.304059    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-credential-socket" pod "11656a86-294a-4cd1-a29c-6f8d02a39c2b" (UID: "11656a86-294a-4cd1-a29c-6f8d02a39c2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.304189 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.304167    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istiod-ca-cert" pod "11656a86-294a-4cd1-a29c-6f8d02a39c2b" (UID: "11656a86-294a-4cd1-a29c-6f8d02a39c2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:22.304549 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.304303    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-podinfo\") pod \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\" (UID: \"11656a86-294a-4cd1-a29c-6f8d02a39c2b\") "
Oct 30 20:10:22.305276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.305118    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.305276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.305148    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df66169-610e-4ff9-8706-25e02d655b48-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.305276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.305163    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.305276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.305179    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df66169-610e-4ff9-8706-25e02d655b48-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.305276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.305194    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wd96\" (UniqueName: \"kubernetes.io/projected/6df66169-610e-4ff9-8706-25e02d655b48-kube-api-access-4wd96\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.305276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.305210    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.305276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.305223    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df66169-610e-4ff9-8706-25e02d655b48-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.305276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.305239    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.305276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.305253    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.308622 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.308597    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11656a86-294a-4cd1-a29c-6f8d02a39c2b-kube-api-access-qgsjj" pod "11656a86-294a-4cd1-a29c-6f8d02a39c2b" (UID: "11656a86-294a-4cd1-a29c-6f8d02a39c2b"). InnerVolumeSpecName "kube-api-access-qgsjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:22.316393 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.316364    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-envoy" pod "11656a86-294a-4cd1-a29c-6f8d02a39c2b" (UID: "11656a86-294a-4cd1-a29c-6f8d02a39c2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:22.325244 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.325213    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-token" pod "11656a86-294a-4cd1-a29c-6f8d02a39c2b" (UID: "11656a86-294a-4cd1-a29c-6f8d02a39c2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:22.325344 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.325248    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-podinfo" pod "11656a86-294a-4cd1-a29c-6f8d02a39c2b" (UID: "11656a86-294a-4cd1-a29c-6f8d02a39c2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:10:22.408042 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.408003    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgsjj\" (UniqueName: \"kubernetes.io/projected/11656a86-294a-4cd1-a29c-6f8d02a39c2b-kube-api-access-qgsjj\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.408042 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.408039    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.408355 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.408055    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.408355 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.408069    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11656a86-294a-4cd1-a29c-6f8d02a39c2b-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:22.443518 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.443487    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"]
Oct 30 20:10:22.455248 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:22.455179    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-fbvzv"]
Oct 30 20:10:23.107170 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:23.107134    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"
Oct 30 20:10:23.107170 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:23.107148    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg" event={"ID":"11656a86-294a-4cd1-a29c-6f8d02a39c2b","Type":"ContainerDied","Data":"ab1c3225299b3f9865ca99aed5bc54fb8b4741878fb10872a6cde357aa08f4ff"}
Oct 30 20:10:23.107720 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:23.107242    4823 scope.go:122] "RemoveContainer" containerID="9fcfd6ff1c9e699ceba6fe0f726cd6e4da9785667e55519cd9e8291e16dca3d6"
Oct 30 20:10:23.110728 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:23.110699    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p" event={"ID":"6df66169-610e-4ff9-8706-25e02d655b48","Type":"ContainerDied","Data":"f2a88e7bc057f07c3511d3ea8a0ff7154544eecbf00f4106fe1351018d119894"}
Oct 30 20:10:23.110914 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:23.110885    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"
Oct 30 20:10:23.121786 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:23.121763    4823 scope.go:122] "RemoveContainer" containerID="9cd9b1c5acdf48457bc4f6b5f47b8f9d7953971c9aad05516ef2e2a1720a46b0"
Oct 30 20:10:23.140673 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:23.140632    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"]
Oct 30 20:10:23.146600 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:23.146576    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-2284p"]
Oct 30 20:10:23.175264 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:23.175236    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"]
Oct 30 20:10:23.192222 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:23.192194    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-4zvvg"]
Oct 30 20:10:23.444778 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:23.444696    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"]
Oct 30 20:10:23.445025 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:23.444994    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz" podUID="4efe25a4-1f89-4f9d-b6f2-bc14c344e69b" containerName="maas-api" containerID="cri-o://69a9f99f527b0514999ce6d899dfc2fdf74d58ce6f3f80ac2814407578955dfc" gracePeriod=30
Oct 30 20:10:23.993847 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:23.992912    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:10:24.123431 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.123399    4823 generic.go:356] "Generic (PLEG): container finished" podID="4efe25a4-1f89-4f9d-b6f2-bc14c344e69b" containerID="69a9f99f527b0514999ce6d899dfc2fdf74d58ce6f3f80ac2814407578955dfc" exitCode=0
Oct 30 20:10:24.123931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.123471    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"
Oct 30 20:10:24.123931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.123501    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz" event={"ID":"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b","Type":"ContainerDied","Data":"69a9f99f527b0514999ce6d899dfc2fdf74d58ce6f3f80ac2814407578955dfc"}
Oct 30 20:10:24.123931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.123556    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz" event={"ID":"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b","Type":"ContainerDied","Data":"536c8547cb5bafe1e95b0f9f00824cf88bb62d647052bccb604e1b75ceb97240"}
Oct 30 20:10:24.123931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.123588    4823 scope.go:122] "RemoveContainer" containerID="69a9f99f527b0514999ce6d899dfc2fdf74d58ce6f3f80ac2814407578955dfc"
Oct 30 20:10:24.135817 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.135789    4823 scope.go:122] "RemoveContainer" containerID="69a9f99f527b0514999ce6d899dfc2fdf74d58ce6f3f80ac2814407578955dfc"
Oct 30 20:10:24.136164 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:10:24.136145    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a9f99f527b0514999ce6d899dfc2fdf74d58ce6f3f80ac2814407578955dfc\": container with ID starting with 69a9f99f527b0514999ce6d899dfc2fdf74d58ce6f3f80ac2814407578955dfc not found: ID does not exist" containerID="69a9f99f527b0514999ce6d899dfc2fdf74d58ce6f3f80ac2814407578955dfc"
Oct 30 20:10:24.136248 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.136175    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a9f99f527b0514999ce6d899dfc2fdf74d58ce6f3f80ac2814407578955dfc"} err="failed to get container status \"69a9f99f527b0514999ce6d899dfc2fdf74d58ce6f3f80ac2814407578955dfc\": rpc error: code = NotFound desc = could not find container \"69a9f99f527b0514999ce6d899dfc2fdf74d58ce6f3f80ac2814407578955dfc\": container with ID starting with 69a9f99f527b0514999ce6d899dfc2fdf74d58ce6f3f80ac2814407578955dfc not found: ID does not exist"
Oct 30 20:10:24.159063 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.159030    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-maas-api-tls\") pod \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\" (UID: \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\") "
Oct 30 20:10:24.159267 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.159106    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-kube-api-access-qnlxh\" (UniqueName: \"kubernetes.io/projected/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-kube-api-access-qnlxh\") pod \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\" (UID: \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\") "
Oct 30 20:10:24.159339 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.159274    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-metrics-certs\") pod \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\" (UID: \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\") "
Oct 30 20:10:24.159339 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.159307    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-openshift-service-ca\") pod \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\" (UID: \"4efe25a4-1f89-4f9d-b6f2-bc14c344e69b\") "
Oct 30 20:10:24.160206 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.159736    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-openshift-service-ca" pod "4efe25a4-1f89-4f9d-b6f2-bc14c344e69b" (UID: "4efe25a4-1f89-4f9d-b6f2-bc14c344e69b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:24.160206 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.160175    4823 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-openshift-service-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:24.169125 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.168260    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-metrics-certs" pod "4efe25a4-1f89-4f9d-b6f2-bc14c344e69b" (UID: "4efe25a4-1f89-4f9d-b6f2-bc14c344e69b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:24.181915 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.181815    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-maas-api-tls" pod "4efe25a4-1f89-4f9d-b6f2-bc14c344e69b" (UID: "4efe25a4-1f89-4f9d-b6f2-bc14c344e69b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:24.186712 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.186676    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-kube-api-access-qnlxh" pod "4efe25a4-1f89-4f9d-b6f2-bc14c344e69b" (UID: "4efe25a4-1f89-4f9d-b6f2-bc14c344e69b"). InnerVolumeSpecName "kube-api-access-qnlxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:24.261565 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.261535    4823 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-metrics-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:24.261565 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.261562    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-maas-api-tls\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:24.261776 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.261582    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnlxh\" (UniqueName: \"kubernetes.io/projected/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b-kube-api-access-qnlxh\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:24.281597 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.281562    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11656a86-294a-4cd1-a29c-6f8d02a39c2b" path="/var/lib/kubelet/pods/11656a86-294a-4cd1-a29c-6f8d02a39c2b/volumes"
Oct 30 20:10:24.282287 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.282256    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc11d6a-c12d-4dab-8356-a8a05ee82f37" path="/var/lib/kubelet/pods/3bc11d6a-c12d-4dab-8356-a8a05ee82f37/volumes"
Oct 30 20:10:24.282837 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.282817    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df66169-610e-4ff9-8706-25e02d655b48" path="/var/lib/kubelet/pods/6df66169-610e-4ff9-8706-25e02d655b48/volumes"
Oct 30 20:10:24.460571 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.460441    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"]
Oct 30 20:10:24.465479 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:24.465449    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-925e04-64c47c7cb9-2d5tz"]
Oct 30 20:10:26.286990 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:26.286947    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4efe25a4-1f89-4f9d-b6f2-bc14c344e69b" path="/var/lib/kubelet/pods/4efe25a4-1f89-4f9d-b6f2-bc14c344e69b/volumes"
Oct 30 20:10:32.023981 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:10:32.023941    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:10:35.520133 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:35.520099    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"]
Oct 30 20:10:35.520621 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:35.520408    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj" podUID="835abdec-0719-4f64-b7e8-4e336b6f12bf" containerName="istio-proxy" containerID="cri-o://3d33cd15521c4ef1ab20786f1104b05523523a7a275ea9567ce794c082fe7900" gracePeriod=30
Oct 30 20:10:40.845326 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.845297    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:10:40.918949 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.918840    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-podinfo\") pod \"835abdec-0719-4f64-b7e8-4e336b6f12bf\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") "
Oct 30 20:10:40.919104 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.919031    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-workload-socket\") pod \"835abdec-0719-4f64-b7e8-4e336b6f12bf\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") "
Oct 30 20:10:40.919144 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.919103    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-data\") pod \"835abdec-0719-4f64-b7e8-4e336b6f12bf\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") "
Oct 30 20:10:40.919144 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.919122    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-workload-certs\") pod \"835abdec-0719-4f64-b7e8-4e336b6f12bf\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") "
Oct 30 20:10:40.919219 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.919162    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-token\" (UniqueName: \"kubernetes.io/projected/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-token\") pod \"835abdec-0719-4f64-b7e8-4e336b6f12bf\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") "
Oct 30 20:10:40.919219 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.919191    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-envoy\") pod \"835abdec-0719-4f64-b7e8-4e336b6f12bf\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") "
Oct 30 20:10:40.919301 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.919259    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835abdec-0719-4f64-b7e8-4e336b6f12bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835abdec-0719-4f64-b7e8-4e336b6f12bf-istiod-ca-cert\") pod \"835abdec-0719-4f64-b7e8-4e336b6f12bf\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") "
Oct 30 20:10:40.919301 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.919281    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-credential-socket\") pod \"835abdec-0719-4f64-b7e8-4e336b6f12bf\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") "
Oct 30 20:10:40.919415 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.919385    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835abdec-0719-4f64-b7e8-4e336b6f12bf-kube-api-access-8k47j\" (UniqueName: \"kubernetes.io/projected/835abdec-0719-4f64-b7e8-4e336b6f12bf-kube-api-access-8k47j\") pod \"835abdec-0719-4f64-b7e8-4e336b6f12bf\" (UID: \"835abdec-0719-4f64-b7e8-4e336b6f12bf\") "
Oct 30 20:10:40.919680 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.919655    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-workload-certs" pod "835abdec-0719-4f64-b7e8-4e336b6f12bf" (UID: "835abdec-0719-4f64-b7e8-4e336b6f12bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:40.919948 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.919932    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:40.920472 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.920441    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-workload-socket" pod "835abdec-0719-4f64-b7e8-4e336b6f12bf" (UID: "835abdec-0719-4f64-b7e8-4e336b6f12bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:40.920586 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.920567    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835abdec-0719-4f64-b7e8-4e336b6f12bf-istiod-ca-cert" pod "835abdec-0719-4f64-b7e8-4e336b6f12bf" (UID: "835abdec-0719-4f64-b7e8-4e336b6f12bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:40.920661 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.920574    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-data" pod "835abdec-0719-4f64-b7e8-4e336b6f12bf" (UID: "835abdec-0719-4f64-b7e8-4e336b6f12bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:40.920717 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.920690    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-credential-socket" pod "835abdec-0719-4f64-b7e8-4e336b6f12bf" (UID: "835abdec-0719-4f64-b7e8-4e336b6f12bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:40.922617 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.922591    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-podinfo" pod "835abdec-0719-4f64-b7e8-4e336b6f12bf" (UID: "835abdec-0719-4f64-b7e8-4e336b6f12bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:10:40.922836 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.922811    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-token" pod "835abdec-0719-4f64-b7e8-4e336b6f12bf" (UID: "835abdec-0719-4f64-b7e8-4e336b6f12bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:40.923126 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.923099    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-envoy" pod "835abdec-0719-4f64-b7e8-4e336b6f12bf" (UID: "835abdec-0719-4f64-b7e8-4e336b6f12bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:40.923674 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:40.923649    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835abdec-0719-4f64-b7e8-4e336b6f12bf-kube-api-access-8k47j" pod "835abdec-0719-4f64-b7e8-4e336b6f12bf" (UID: "835abdec-0719-4f64-b7e8-4e336b6f12bf"). InnerVolumeSpecName "kube-api-access-8k47j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:41.021143 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.021109    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835abdec-0719-4f64-b7e8-4e336b6f12bf-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:41.021143 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.021145    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:41.021340 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.021156    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k47j\" (UniqueName: \"kubernetes.io/projected/835abdec-0719-4f64-b7e8-4e336b6f12bf-kube-api-access-8k47j\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:41.021340 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.021166    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:41.021340 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.021175    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:41.021340 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.021184    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:41.021340 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.021193    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:41.021340 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.021202    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835abdec-0719-4f64-b7e8-4e336b6f12bf-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:41.339238 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.339204    4823 generic.go:356] "Generic (PLEG): container finished" podID="835abdec-0719-4f64-b7e8-4e336b6f12bf" containerID="3d33cd15521c4ef1ab20786f1104b05523523a7a275ea9567ce794c082fe7900" exitCode=0
Oct 30 20:10:41.339443 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.339269    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"
Oct 30 20:10:41.339443 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.339365    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj" event={"ID":"835abdec-0719-4f64-b7e8-4e336b6f12bf","Type":"ContainerDied","Data":"3d33cd15521c4ef1ab20786f1104b05523523a7a275ea9567ce794c082fe7900"}
Oct 30 20:10:41.339443 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.339429    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj" event={"ID":"835abdec-0719-4f64-b7e8-4e336b6f12bf","Type":"ContainerDied","Data":"2e4d9fab465ebb8ef1892b393c97e71f94614ddc9d8b32ffe59dc1e2c0755aad"}
Oct 30 20:10:41.339581 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.339463    4823 scope.go:122] "RemoveContainer" containerID="3d33cd15521c4ef1ab20786f1104b05523523a7a275ea9567ce794c082fe7900"
Oct 30 20:10:41.356127 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.356105    4823 scope.go:122] "RemoveContainer" containerID="b8cb4e161b5701d775fc397a00ff0cc8ed987ace4598c454879f20e69f1c0631"
Oct 30 20:10:41.368118 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.368097    4823 scope.go:122] "RemoveContainer" containerID="3d33cd15521c4ef1ab20786f1104b05523523a7a275ea9567ce794c082fe7900"
Oct 30 20:10:41.368451 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:10:41.368426    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d33cd15521c4ef1ab20786f1104b05523523a7a275ea9567ce794c082fe7900\": container with ID starting with 3d33cd15521c4ef1ab20786f1104b05523523a7a275ea9567ce794c082fe7900 not found: ID does not exist" containerID="3d33cd15521c4ef1ab20786f1104b05523523a7a275ea9567ce794c082fe7900"
Oct 30 20:10:41.368516 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.368460    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d33cd15521c4ef1ab20786f1104b05523523a7a275ea9567ce794c082fe7900"} err="failed to get container status \"3d33cd15521c4ef1ab20786f1104b05523523a7a275ea9567ce794c082fe7900\": rpc error: code = NotFound desc = could not find container \"3d33cd15521c4ef1ab20786f1104b05523523a7a275ea9567ce794c082fe7900\": container with ID starting with 3d33cd15521c4ef1ab20786f1104b05523523a7a275ea9567ce794c082fe7900 not found: ID does not exist"
Oct 30 20:10:41.368516 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.368477    4823 scope.go:122] "RemoveContainer" containerID="b8cb4e161b5701d775fc397a00ff0cc8ed987ace4598c454879f20e69f1c0631"
Oct 30 20:10:41.368764 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:10:41.368745    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8cb4e161b5701d775fc397a00ff0cc8ed987ace4598c454879f20e69f1c0631\": container with ID starting with b8cb4e161b5701d775fc397a00ff0cc8ed987ace4598c454879f20e69f1c0631 not found: ID does not exist" containerID="b8cb4e161b5701d775fc397a00ff0cc8ed987ace4598c454879f20e69f1c0631"
Oct 30 20:10:41.368824 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.368774    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8cb4e161b5701d775fc397a00ff0cc8ed987ace4598c454879f20e69f1c0631"} err="failed to get container status \"b8cb4e161b5701d775fc397a00ff0cc8ed987ace4598c454879f20e69f1c0631\": rpc error: code = NotFound desc = could not find container \"b8cb4e161b5701d775fc397a00ff0cc8ed987ace4598c454879f20e69f1c0631\": container with ID starting with b8cb4e161b5701d775fc397a00ff0cc8ed987ace4598c454879f20e69f1c0631 not found: ID does not exist"
Oct 30 20:10:41.384099 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.384066    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"]
Oct 30 20:10:41.389347 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:41.389321    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-925e04-openshift-default-6db84576f-mdkfj"]
Oct 30 20:10:42.282011 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:42.281981    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="835abdec-0719-4f64-b7e8-4e336b6f12bf" path="/var/lib/kubelet/pods/835abdec-0719-4f64-b7e8-4e336b6f12bf/volumes"
Oct 30 20:10:44.759568 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:44.759532    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"]
Oct 30 20:10:44.760102 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:44.759821    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq" podUID="20f67270-49ff-4e06-988f-84deb2909f8e" containerName="maas-api" containerID="cri-o://21eb596155b1ad11f0795305861c3330667ba6deef35d542821a90816c38b0cd" gracePeriod=30
Oct 30 20:10:44.863895 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:44.863817    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq" podUID="20f67270-49ff-4e06-988f-84deb2909f8e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.182:8443/health\": dial tcp 10.132.0.182:8443: connect: connection refused"
Oct 30 20:10:45.354883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.351463    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:10:45.396734 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.396697    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20f67270-49ff-4e06-988f-84deb2909f8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f67270-49ff-4e06-988f-84deb2909f8e-metrics-certs\") pod \"20f67270-49ff-4e06-988f-84deb2909f8e\" (UID: \"20f67270-49ff-4e06-988f-84deb2909f8e\") "
Oct 30 20:10:45.396989 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.396761    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f67270-49ff-4e06-988f-84deb2909f8e-kube-api-access-jnzt5\" (UniqueName: \"kubernetes.io/projected/20f67270-49ff-4e06-988f-84deb2909f8e-kube-api-access-jnzt5\") pod \"20f67270-49ff-4e06-988f-84deb2909f8e\" (UID: \"20f67270-49ff-4e06-988f-84deb2909f8e\") "
Oct 30 20:10:45.396989 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.396820    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20f67270-49ff-4e06-988f-84deb2909f8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f67270-49ff-4e06-988f-84deb2909f8e-maas-api-tls\") pod \"20f67270-49ff-4e06-988f-84deb2909f8e\" (UID: \"20f67270-49ff-4e06-988f-84deb2909f8e\") "
Oct 30 20:10:45.396989 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.396961    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f67270-49ff-4e06-988f-84deb2909f8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f67270-49ff-4e06-988f-84deb2909f8e-openshift-service-ca\") pod \"20f67270-49ff-4e06-988f-84deb2909f8e\" (UID: \"20f67270-49ff-4e06-988f-84deb2909f8e\") "
Oct 30 20:10:45.399972 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.399517    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f67270-49ff-4e06-988f-84deb2909f8e-openshift-service-ca" pod "20f67270-49ff-4e06-988f-84deb2909f8e" (UID: "20f67270-49ff-4e06-988f-84deb2909f8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:45.402973 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.402919    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f67270-49ff-4e06-988f-84deb2909f8e-metrics-certs" pod "20f67270-49ff-4e06-988f-84deb2909f8e" (UID: "20f67270-49ff-4e06-988f-84deb2909f8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:45.405022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.404991    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f67270-49ff-4e06-988f-84deb2909f8e-maas-api-tls" pod "20f67270-49ff-4e06-988f-84deb2909f8e" (UID: "20f67270-49ff-4e06-988f-84deb2909f8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:45.406109 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.406078    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f67270-49ff-4e06-988f-84deb2909f8e-kube-api-access-jnzt5" pod "20f67270-49ff-4e06-988f-84deb2909f8e" (UID: "20f67270-49ff-4e06-988f-84deb2909f8e"). InnerVolumeSpecName "kube-api-access-jnzt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:45.406794 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.406763    4823 generic.go:356] "Generic (PLEG): container finished" podID="20f67270-49ff-4e06-988f-84deb2909f8e" containerID="21eb596155b1ad11f0795305861c3330667ba6deef35d542821a90816c38b0cd" exitCode=0
Oct 30 20:10:45.407069 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.407044    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq" event={"ID":"20f67270-49ff-4e06-988f-84deb2909f8e","Type":"ContainerDied","Data":"21eb596155b1ad11f0795305861c3330667ba6deef35d542821a90816c38b0cd"}
Oct 30 20:10:45.407152 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.407104    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq" event={"ID":"20f67270-49ff-4e06-988f-84deb2909f8e","Type":"ContainerDied","Data":"8f6f2db01433ddfe840c1570906e2d0d03113d6286af2ae682dec00bafac2cab"}
Oct 30 20:10:45.407152 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.407138    4823 scope.go:122] "RemoveContainer" containerID="21eb596155b1ad11f0795305861c3330667ba6deef35d542821a90816c38b0cd"
Oct 30 20:10:45.407299 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.407281    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"
Oct 30 20:10:45.427519 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.427497    4823 scope.go:122] "RemoveContainer" containerID="21eb596155b1ad11f0795305861c3330667ba6deef35d542821a90816c38b0cd"
Oct 30 20:10:45.427996 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:10:45.427968    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21eb596155b1ad11f0795305861c3330667ba6deef35d542821a90816c38b0cd\": container with ID starting with 21eb596155b1ad11f0795305861c3330667ba6deef35d542821a90816c38b0cd not found: ID does not exist" containerID="21eb596155b1ad11f0795305861c3330667ba6deef35d542821a90816c38b0cd"
Oct 30 20:10:45.428089 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.428008    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21eb596155b1ad11f0795305861c3330667ba6deef35d542821a90816c38b0cd"} err="failed to get container status \"21eb596155b1ad11f0795305861c3330667ba6deef35d542821a90816c38b0cd\": rpc error: code = NotFound desc = could not find container \"21eb596155b1ad11f0795305861c3330667ba6deef35d542821a90816c38b0cd\": container with ID starting with 21eb596155b1ad11f0795305861c3330667ba6deef35d542821a90816c38b0cd not found: ID does not exist"
Oct 30 20:10:45.445122 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.445098    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"]
Oct 30 20:10:45.455285 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.455259    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7f7d21-678fbb6d55-tvssq"]
Oct 30 20:10:45.499414 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.499384    4823 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f67270-49ff-4e06-988f-84deb2909f8e-metrics-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:45.499414 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.499415    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnzt5\" (UniqueName: \"kubernetes.io/projected/20f67270-49ff-4e06-988f-84deb2909f8e-kube-api-access-jnzt5\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:45.499611 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.499426    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f67270-49ff-4e06-988f-84deb2909f8e-maas-api-tls\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:45.499611 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:45.499447    4823 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f67270-49ff-4e06-988f-84deb2909f8e-openshift-service-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:46.284584 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:46.284544    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f67270-49ff-4e06-988f-84deb2909f8e" path="/var/lib/kubelet/pods/20f67270-49ff-4e06-988f-84deb2909f8e/volumes"
Oct 30 20:10:56.055669 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.055631    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"]
Oct 30 20:10:56.056315 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.055908    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556" podUID="d0c0ca13-508b-45d0-ade5-75dbf9bbde50" containerName="maas-api" containerID="cri-o://357e1945030cb62d7c680dafbbd3965ac0bff9269fc23031dbf29421b467fd6b" gracePeriod=30
Oct 30 20:10:56.676166 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.672046    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:10:56.676166 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.674176    4823 generic.go:356] "Generic (PLEG): container finished" podID="d0c0ca13-508b-45d0-ade5-75dbf9bbde50" containerID="357e1945030cb62d7c680dafbbd3965ac0bff9269fc23031dbf29421b467fd6b" exitCode=0
Oct 30 20:10:56.676166 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.674274    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556" event={"ID":"d0c0ca13-508b-45d0-ade5-75dbf9bbde50","Type":"ContainerDied","Data":"357e1945030cb62d7c680dafbbd3965ac0bff9269fc23031dbf29421b467fd6b"}
Oct 30 20:10:56.676166 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.674329    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556" event={"ID":"d0c0ca13-508b-45d0-ade5-75dbf9bbde50","Type":"ContainerDied","Data":"18646ef99c9f7c9c81fc8832eddef0c3211943f6fc57f5d2e5454c62f02dd2fa"}
Oct 30 20:10:56.676166 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.674362    4823 scope.go:122] "RemoveContainer" containerID="357e1945030cb62d7c680dafbbd3965ac0bff9269fc23031dbf29421b467fd6b"
Oct 30 20:10:56.709519 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.709493    4823 scope.go:122] "RemoveContainer" containerID="357e1945030cb62d7c680dafbbd3965ac0bff9269fc23031dbf29421b467fd6b"
Oct 30 20:10:56.713144 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:10:56.713107    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357e1945030cb62d7c680dafbbd3965ac0bff9269fc23031dbf29421b467fd6b\": container with ID starting with 357e1945030cb62d7c680dafbbd3965ac0bff9269fc23031dbf29421b467fd6b not found: ID does not exist" containerID="357e1945030cb62d7c680dafbbd3965ac0bff9269fc23031dbf29421b467fd6b"
Oct 30 20:10:56.713286 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.713154    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357e1945030cb62d7c680dafbbd3965ac0bff9269fc23031dbf29421b467fd6b"} err="failed to get container status \"357e1945030cb62d7c680dafbbd3965ac0bff9269fc23031dbf29421b467fd6b\": rpc error: code = NotFound desc = could not find container \"357e1945030cb62d7c680dafbbd3965ac0bff9269fc23031dbf29421b467fd6b\": container with ID starting with 357e1945030cb62d7c680dafbbd3965ac0bff9269fc23031dbf29421b467fd6b not found: ID does not exist"
Oct 30 20:10:56.863948 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.854644    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-metrics-certs\") pod \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\" (UID: \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\") "
Oct 30 20:10:56.863948 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.854758    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-kube-api-access-kv9x9\" (UniqueName: \"kubernetes.io/projected/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-kube-api-access-kv9x9\") pod \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\" (UID: \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\") "
Oct 30 20:10:56.863948 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.854967    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-openshift-service-ca\") pod \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\" (UID: \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\") "
Oct 30 20:10:56.863948 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.855033    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-maas-api-tls\") pod \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\" (UID: \"d0c0ca13-508b-45d0-ade5-75dbf9bbde50\") "
Oct 30 20:10:56.886014 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.876272    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-openshift-service-ca" pod "d0c0ca13-508b-45d0-ade5-75dbf9bbde50" (UID: "d0c0ca13-508b-45d0-ade5-75dbf9bbde50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:56.903111 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.901713    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-maas-api-tls" pod "d0c0ca13-508b-45d0-ade5-75dbf9bbde50" (UID: "d0c0ca13-508b-45d0-ade5-75dbf9bbde50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:56.903111 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.901823    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-metrics-certs" pod "d0c0ca13-508b-45d0-ade5-75dbf9bbde50" (UID: "d0c0ca13-508b-45d0-ade5-75dbf9bbde50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:56.911570 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.910066    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-kube-api-access-kv9x9" pod "d0c0ca13-508b-45d0-ade5-75dbf9bbde50" (UID: "d0c0ca13-508b-45d0-ade5-75dbf9bbde50"). InnerVolumeSpecName "kube-api-access-kv9x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:56.972480 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.972445    4823 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-metrics-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:56.972480 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.972477    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv9x9\" (UniqueName: \"kubernetes.io/projected/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-kube-api-access-kv9x9\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:56.972708 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.972490    4823 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-openshift-service-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:56.972708 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:56.972501    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0c0ca13-508b-45d0-ade5-75dbf9bbde50-maas-api-tls\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:57.684011 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:57.683976    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"
Oct 30 20:10:57.713942 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:57.713907    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"]
Oct 30 20:10:57.718417 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:57.718390    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f246b-5b954bfbcc-w9556"]
Oct 30 20:10:58.285353 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:58.285316    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c0ca13-508b-45d0-ade5-75dbf9bbde50" path="/var/lib/kubelet/pods/d0c0ca13-508b-45d0-ade5-75dbf9bbde50/volumes"
Oct 30 20:10:59.344483 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.344442    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"]
Oct 30 20:10:59.345430 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345407    4823 cpu_manager.go:414] "containerMap: removing container" podUID="6df66169-610e-4ff9-8706-25e02d655b48" containerName="istio-proxy"
Oct 30 20:10:59.345430 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345430    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df66169-610e-4ff9-8706-25e02d655b48" containerName="istio-proxy"
Oct 30 20:10:59.345602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345470    4823 cpu_manager.go:414] "containerMap: removing container" podUID="20f67270-49ff-4e06-988f-84deb2909f8e" containerName="maas-api"
Oct 30 20:10:59.345602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345478    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f67270-49ff-4e06-988f-84deb2909f8e" containerName="maas-api"
Oct 30 20:10:59.345602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345499    4823 cpu_manager.go:414] "containerMap: removing container" podUID="d0c0ca13-508b-45d0-ade5-75dbf9bbde50" containerName="maas-api"
Oct 30 20:10:59.345602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345507    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c0ca13-508b-45d0-ade5-75dbf9bbde50" containerName="maas-api"
Oct 30 20:10:59.345602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345527    4823 cpu_manager.go:414] "containerMap: removing container" podUID="11656a86-294a-4cd1-a29c-6f8d02a39c2b" containerName="istio-proxy"
Oct 30 20:10:59.345602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345536    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11656a86-294a-4cd1-a29c-6f8d02a39c2b" containerName="istio-proxy"
Oct 30 20:10:59.345602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345560    4823 cpu_manager.go:414] "containerMap: removing container" podUID="be418f6d-e900-479d-bbab-3b94ffba7a7f" containerName="discovery"
Oct 30 20:10:59.345602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345569    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be418f6d-e900-479d-bbab-3b94ffba7a7f" containerName="discovery"
Oct 30 20:10:59.345602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345587    4823 cpu_manager.go:414] "containerMap: removing container" podUID="835abdec-0719-4f64-b7e8-4e336b6f12bf" containerName="istio-proxy"
Oct 30 20:10:59.345602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345594    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835abdec-0719-4f64-b7e8-4e336b6f12bf" containerName="istio-proxy"
Oct 30 20:10:59.346182 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345612    4823 cpu_manager.go:414] "containerMap: removing container" podUID="3bc11d6a-c12d-4dab-8356-a8a05ee82f37" containerName="istio-proxy"
Oct 30 20:10:59.346182 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345619    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc11d6a-c12d-4dab-8356-a8a05ee82f37" containerName="istio-proxy"
Oct 30 20:10:59.346182 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345633    4823 cpu_manager.go:414] "containerMap: removing container" podUID="4efe25a4-1f89-4f9d-b6f2-bc14c344e69b" containerName="maas-api"
Oct 30 20:10:59.346182 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345641    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efe25a4-1f89-4f9d-b6f2-bc14c344e69b" containerName="maas-api"
Oct 30 20:10:59.346182 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345836    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f67270-49ff-4e06-988f-84deb2909f8e" containerName="maas-api"
Oct 30 20:10:59.346182 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345851    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c0ca13-508b-45d0-ade5-75dbf9bbde50" containerName="maas-api"
Oct 30 20:10:59.346182 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345882    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="11656a86-294a-4cd1-a29c-6f8d02a39c2b" containerName="istio-proxy"
Oct 30 20:10:59.346182 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345894    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="be418f6d-e900-479d-bbab-3b94ffba7a7f" containerName="discovery"
Oct 30 20:10:59.346182 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345902    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="835abdec-0719-4f64-b7e8-4e336b6f12bf" containerName="istio-proxy"
Oct 30 20:10:59.346182 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345912    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc11d6a-c12d-4dab-8356-a8a05ee82f37" containerName="istio-proxy"
Oct 30 20:10:59.346182 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345922    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efe25a4-1f89-4f9d-b6f2-bc14c344e69b" containerName="maas-api"
Oct 30 20:10:59.346182 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.345936    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df66169-610e-4ff9-8706-25e02d655b48" containerName="istio-proxy"
Oct 30 20:10:59.351415 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.351392    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"
Oct 30 20:10:59.362201 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.362141    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"]
Oct 30 20:10:59.409759 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.409725    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:10:59.409996 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.409903    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:10:59.418341 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.418305    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef12d45-3357-4f21-9d3e-3d073a82cc3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1685160c-7l8cb\" (UID: \"eef12d45-3357-4f21-9d3e-3d073a82cc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"
Oct 30 20:10:59.418519 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.418353    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzvj8\" (UniqueName: \"kubernetes.io/projected/eef12d45-3357-4f21-9d3e-3d073a82cc3b-kube-api-access-vzvj8\") pod \"maas-api-revoke-sub-1685160c-7l8cb\" (UID: \"eef12d45-3357-4f21-9d3e-3d073a82cc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"
Oct 30 20:10:59.418590 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.418514    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef12d45-3357-4f21-9d3e-3d073a82cc3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1685160c-7l8cb\" (UID: \"eef12d45-3357-4f21-9d3e-3d073a82cc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"
Oct 30 20:10:59.520502 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.520463    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef12d45-3357-4f21-9d3e-3d073a82cc3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1685160c-7l8cb\" (UID: \"eef12d45-3357-4f21-9d3e-3d073a82cc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"
Oct 30 20:10:59.520502 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.520503    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzvj8\" (UniqueName: \"kubernetes.io/projected/eef12d45-3357-4f21-9d3e-3d073a82cc3b-kube-api-access-vzvj8\") pod \"maas-api-revoke-sub-1685160c-7l8cb\" (UID: \"eef12d45-3357-4f21-9d3e-3d073a82cc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"
Oct 30 20:10:59.520774 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.520645    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef12d45-3357-4f21-9d3e-3d073a82cc3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1685160c-7l8cb\" (UID: \"eef12d45-3357-4f21-9d3e-3d073a82cc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"
Oct 30 20:10:59.521332 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.521309    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef12d45-3357-4f21-9d3e-3d073a82cc3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1685160c-7l8cb\" (UID: \"eef12d45-3357-4f21-9d3e-3d073a82cc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"
Oct 30 20:10:59.523027 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.523005    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef12d45-3357-4f21-9d3e-3d073a82cc3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1685160c-7l8cb\" (UID: \"eef12d45-3357-4f21-9d3e-3d073a82cc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"
Oct 30 20:10:59.533194 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.533171    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzvj8\" (UniqueName: \"kubernetes.io/projected/eef12d45-3357-4f21-9d3e-3d073a82cc3b-kube-api-access-vzvj8\") pod \"maas-api-revoke-sub-1685160c-7l8cb\" (UID: \"eef12d45-3357-4f21-9d3e-3d073a82cc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"
Oct 30 20:10:59.676901 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.676836    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"
Oct 30 20:10:59.979651 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:10:59.979622    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"]
Oct 30 20:10:59.981848 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:10:59.981818    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef12d45_3357_4f21_9d3e_3d073a82cc3b.slice/crio-47d70a9faecd54ba48955046b3a3737272e15551df7e96695116a77c24238adb WatchSource:0}: Error finding container 47d70a9faecd54ba48955046b3a3737272e15551df7e96695116a77c24238adb: Status 404 returned error can't find the container with id 47d70a9faecd54ba48955046b3a3737272e15551df7e96695116a77c24238adb
Oct 30 20:11:00.500328 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.500292    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"]
Oct 30 20:11:00.501114 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.501097    4823 cpu_manager.go:414] "containerMap: removing container" podUID="835abdec-0719-4f64-b7e8-4e336b6f12bf" containerName="istio-proxy"
Oct 30 20:11:00.501179 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.501115    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835abdec-0719-4f64-b7e8-4e336b6f12bf" containerName="istio-proxy"
Oct 30 20:11:00.501272 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.501263    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="835abdec-0719-4f64-b7e8-4e336b6f12bf" containerName="istio-proxy"
Oct 30 20:11:00.505076 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.505054    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"
Oct 30 20:11:00.514918 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.514884    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"]
Oct 30 20:11:00.537515 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.537427    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26e56c15-b87d-4b0a-b403-f32809781e5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-889acd9b-9qsdq\" (UID: \"26e56c15-b87d-4b0a-b403-f32809781e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"
Oct 30 20:11:00.537685 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.537601    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdvn\" (UniqueName: \"kubernetes.io/projected/26e56c15-b87d-4b0a-b403-f32809781e5d-kube-api-access-ctdvn\") pod \"maas-api-revoke-sub-889acd9b-9qsdq\" (UID: \"26e56c15-b87d-4b0a-b403-f32809781e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"
Oct 30 20:11:00.537844 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.537828    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26e56c15-b87d-4b0a-b403-f32809781e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-889acd9b-9qsdq\" (UID: \"26e56c15-b87d-4b0a-b403-f32809781e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"
Oct 30 20:11:00.639784 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.639752    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26e56c15-b87d-4b0a-b403-f32809781e5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-889acd9b-9qsdq\" (UID: \"26e56c15-b87d-4b0a-b403-f32809781e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"
Oct 30 20:11:00.639967 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.639844    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdvn\" (UniqueName: \"kubernetes.io/projected/26e56c15-b87d-4b0a-b403-f32809781e5d-kube-api-access-ctdvn\") pod \"maas-api-revoke-sub-889acd9b-9qsdq\" (UID: \"26e56c15-b87d-4b0a-b403-f32809781e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"
Oct 30 20:11:00.640090 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.640072    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26e56c15-b87d-4b0a-b403-f32809781e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-889acd9b-9qsdq\" (UID: \"26e56c15-b87d-4b0a-b403-f32809781e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"
Oct 30 20:11:00.640687 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.640669    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26e56c15-b87d-4b0a-b403-f32809781e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-889acd9b-9qsdq\" (UID: \"26e56c15-b87d-4b0a-b403-f32809781e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"
Oct 30 20:11:00.642277 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.642256    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26e56c15-b87d-4b0a-b403-f32809781e5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-889acd9b-9qsdq\" (UID: \"26e56c15-b87d-4b0a-b403-f32809781e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"
Oct 30 20:11:00.651791 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.651761    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdvn\" (UniqueName: \"kubernetes.io/projected/26e56c15-b87d-4b0a-b403-f32809781e5d-kube-api-access-ctdvn\") pod \"maas-api-revoke-sub-889acd9b-9qsdq\" (UID: \"26e56c15-b87d-4b0a-b403-f32809781e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"
Oct 30 20:11:00.714759 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.714727    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb" event={"ID":"eef12d45-3357-4f21-9d3e-3d073a82cc3b","Type":"ContainerStarted","Data":"24ac747c30a5d0219a4e15b2c9e1e4ab9b34a40a64630ed0722c05a0fe34f451"}
Oct 30 20:11:00.714962 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.714778    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb" event={"ID":"eef12d45-3357-4f21-9d3e-3d073a82cc3b","Type":"ContainerStarted","Data":"47d70a9faecd54ba48955046b3a3737272e15551df7e96695116a77c24238adb"}
Oct 30 20:11:00.739892 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.739811    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb" podStartSLOduration=1.739793009 podStartE2EDuration="1.739793009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:00.735624422 +0000 UTC m=+2460.628238746" watchObservedRunningTime="2026-10-30 20:11:00.739793009 +0000 UTC m=+2460.632407335"
Oct 30 20:11:00.825412 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:00.825325    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"
Oct 30 20:11:01.160958 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:11:01.160927    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e56c15_b87d_4b0a_b403_f32809781e5d.slice/crio-4f75c7bbf82eceee60876a15c19b2b12648b34a4703b7f28d41642e1c10bc5f9 WatchSource:0}: Error finding container 4f75c7bbf82eceee60876a15c19b2b12648b34a4703b7f28d41642e1c10bc5f9: Status 404 returned error can't find the container with id 4f75c7bbf82eceee60876a15c19b2b12648b34a4703b7f28d41642e1c10bc5f9
Oct 30 20:11:01.161559 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:01.161487    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"]
Oct 30 20:11:01.732336 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:01.732305    4823 generic.go:356] "Generic (PLEG): container finished" podID="eef12d45-3357-4f21-9d3e-3d073a82cc3b" containerID="24ac747c30a5d0219a4e15b2c9e1e4ab9b34a40a64630ed0722c05a0fe34f451" exitCode=0
Oct 30 20:11:01.732823 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:01.732393    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb" event={"ID":"eef12d45-3357-4f21-9d3e-3d073a82cc3b","Type":"ContainerDied","Data":"24ac747c30a5d0219a4e15b2c9e1e4ab9b34a40a64630ed0722c05a0fe34f451"}
Oct 30 20:11:01.734150 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:01.734122    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq" event={"ID":"26e56c15-b87d-4b0a-b403-f32809781e5d","Type":"ContainerStarted","Data":"01eb318c412caf5bed774300b3cdaa9adc16f80e8f65128b0be28c602ee120ef"}
Oct 30 20:11:01.734271 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:01.734180    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq" event={"ID":"26e56c15-b87d-4b0a-b403-f32809781e5d","Type":"ContainerStarted","Data":"4f75c7bbf82eceee60876a15c19b2b12648b34a4703b7f28d41642e1c10bc5f9"}
Oct 30 20:11:01.775142 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:01.775091    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq" podStartSLOduration=1.775078575 podStartE2EDuration="1.775078575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:01.77435058 +0000 UTC m=+2461.666964904" watchObservedRunningTime="2026-10-30 20:11:01.775078575 +0000 UTC m=+2461.667692898"
Oct 30 20:11:02.016679 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:11:02.016638    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:11:02.397138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:02.397096    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"]
Oct 30 20:11:02.397434 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:02.397350    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd" podUID="9dfba44f-22b1-4336-8938-b29d8478811f" containerName="maas-api" containerID="cri-o://d87e9b6a7434c574b39820d760b36e4c8a256411b8d8094015e556907f00cce7" gracePeriod=30
Oct 30 20:11:03.028334 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.028304    4823 generic.go:356] "Generic (PLEG): container finished" podID="9dfba44f-22b1-4336-8938-b29d8478811f" containerID="d87e9b6a7434c574b39820d760b36e4c8a256411b8d8094015e556907f00cce7" exitCode=0
Oct 30 20:11:03.028752 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.028346    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd" event={"ID":"9dfba44f-22b1-4336-8938-b29d8478811f","Type":"ContainerDied","Data":"d87e9b6a7434c574b39820d760b36e4c8a256411b8d8094015e556907f00cce7"}
Oct 30 20:11:03.030697 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.030670    4823 generic.go:356] "Generic (PLEG): container finished" podID="26e56c15-b87d-4b0a-b403-f32809781e5d" containerID="01eb318c412caf5bed774300b3cdaa9adc16f80e8f65128b0be28c602ee120ef" exitCode=0
Oct 30 20:11:03.030820 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.030749    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq" event={"ID":"26e56c15-b87d-4b0a-b403-f32809781e5d","Type":"ContainerDied","Data":"01eb318c412caf5bed774300b3cdaa9adc16f80e8f65128b0be28c602ee120ef"}
Oct 30 20:11:03.032442 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.032421    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:11:03.182399 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.182360    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dfba44f-22b1-4336-8938-b29d8478811f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfba44f-22b1-4336-8938-b29d8478811f-openshift-service-ca\") pod \"9dfba44f-22b1-4336-8938-b29d8478811f\" (UID: \"9dfba44f-22b1-4336-8938-b29d8478811f\") "
Oct 30 20:11:03.182602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.182512    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dfba44f-22b1-4336-8938-b29d8478811f-kube-api-access-k67cc\" (UniqueName: \"kubernetes.io/projected/9dfba44f-22b1-4336-8938-b29d8478811f-kube-api-access-k67cc\") pod \"9dfba44f-22b1-4336-8938-b29d8478811f\" (UID: \"9dfba44f-22b1-4336-8938-b29d8478811f\") "
Oct 30 20:11:03.182602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.182577    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dfba44f-22b1-4336-8938-b29d8478811f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dfba44f-22b1-4336-8938-b29d8478811f-metrics-certs\") pod \"9dfba44f-22b1-4336-8938-b29d8478811f\" (UID: \"9dfba44f-22b1-4336-8938-b29d8478811f\") "
Oct 30 20:11:03.182765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.182673    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dfba44f-22b1-4336-8938-b29d8478811f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dfba44f-22b1-4336-8938-b29d8478811f-maas-api-tls\") pod \"9dfba44f-22b1-4336-8938-b29d8478811f\" (UID: \"9dfba44f-22b1-4336-8938-b29d8478811f\") "
Oct 30 20:11:03.183497 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.183463    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dfba44f-22b1-4336-8938-b29d8478811f-openshift-service-ca" pod "9dfba44f-22b1-4336-8938-b29d8478811f" (UID: "9dfba44f-22b1-4336-8938-b29d8478811f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:03.183744 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.183720    4823 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfba44f-22b1-4336-8938-b29d8478811f-openshift-service-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:03.185899 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.185843    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dfba44f-22b1-4336-8938-b29d8478811f-metrics-certs" pod "9dfba44f-22b1-4336-8938-b29d8478811f" (UID: "9dfba44f-22b1-4336-8938-b29d8478811f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:03.186446 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.186423    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dfba44f-22b1-4336-8938-b29d8478811f-kube-api-access-k67cc" pod "9dfba44f-22b1-4336-8938-b29d8478811f" (UID: "9dfba44f-22b1-4336-8938-b29d8478811f"). InnerVolumeSpecName "kube-api-access-k67cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:03.195367 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.195334    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dfba44f-22b1-4336-8938-b29d8478811f-maas-api-tls" pod "9dfba44f-22b1-4336-8938-b29d8478811f" (UID: "9dfba44f-22b1-4336-8938-b29d8478811f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:03.271879 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.271842    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"
Oct 30 20:11:03.285439 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.285412    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k67cc\" (UniqueName: \"kubernetes.io/projected/9dfba44f-22b1-4336-8938-b29d8478811f-kube-api-access-k67cc\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:03.285439 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.285438    4823 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dfba44f-22b1-4336-8938-b29d8478811f-metrics-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:03.285609 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.285448    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dfba44f-22b1-4336-8938-b29d8478811f-maas-api-tls\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:03.420074 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.419984    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eef12d45-3357-4f21-9d3e-3d073a82cc3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef12d45-3357-4f21-9d3e-3d073a82cc3b-maas-api-openshift-ca\") pod \"eef12d45-3357-4f21-9d3e-3d073a82cc3b\" (UID: \"eef12d45-3357-4f21-9d3e-3d073a82cc3b\") "
Oct 30 20:11:03.420240 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.420219    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eef12d45-3357-4f21-9d3e-3d073a82cc3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef12d45-3357-4f21-9d3e-3d073a82cc3b-maas-api-xks-ca\") pod \"eef12d45-3357-4f21-9d3e-3d073a82cc3b\" (UID: \"eef12d45-3357-4f21-9d3e-3d073a82cc3b\") "
Oct 30 20:11:03.420429 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.420409    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef12d45-3357-4f21-9d3e-3d073a82cc3b-kube-api-access-vzvj8\" (UniqueName: \"kubernetes.io/projected/eef12d45-3357-4f21-9d3e-3d073a82cc3b-kube-api-access-vzvj8\") pod \"eef12d45-3357-4f21-9d3e-3d073a82cc3b\" (UID: \"eef12d45-3357-4f21-9d3e-3d073a82cc3b\") "
Oct 30 20:11:03.421557 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.421506    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef12d45-3357-4f21-9d3e-3d073a82cc3b-maas-api-openshift-ca" pod "eef12d45-3357-4f21-9d3e-3d073a82cc3b" (UID: "eef12d45-3357-4f21-9d3e-3d073a82cc3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:03.425427 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.425395    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef12d45-3357-4f21-9d3e-3d073a82cc3b-kube-api-access-vzvj8" pod "eef12d45-3357-4f21-9d3e-3d073a82cc3b" (UID: "eef12d45-3357-4f21-9d3e-3d073a82cc3b"). InnerVolumeSpecName "kube-api-access-vzvj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:03.427333 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.427303    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef12d45-3357-4f21-9d3e-3d073a82cc3b-maas-api-xks-ca" pod "eef12d45-3357-4f21-9d3e-3d073a82cc3b" (UID: "eef12d45-3357-4f21-9d3e-3d073a82cc3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:03.523282 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.523239    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzvj8\" (UniqueName: \"kubernetes.io/projected/eef12d45-3357-4f21-9d3e-3d073a82cc3b-kube-api-access-vzvj8\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:03.523282 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.523276    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef12d45-3357-4f21-9d3e-3d073a82cc3b-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:03.523495 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:03.523295    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef12d45-3357-4f21-9d3e-3d073a82cc3b-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:04.040267 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.040229    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"
Oct 30 20:11:04.040267 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.040257    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb" event={"ID":"eef12d45-3357-4f21-9d3e-3d073a82cc3b","Type":"ContainerDied","Data":"47d70a9faecd54ba48955046b3a3737272e15551df7e96695116a77c24238adb"}
Oct 30 20:11:04.040806 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.040306    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47d70a9faecd54ba48955046b3a3737272e15551df7e96695116a77c24238adb"
Oct 30 20:11:04.042002 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.041984    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"
Oct 30 20:11:04.042110 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.042029    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd" event={"ID":"9dfba44f-22b1-4336-8938-b29d8478811f","Type":"ContainerDied","Data":"f6384ee1353922c61815000810965e0d4cd691b687e579c7a4689833340d16ca"}
Oct 30 20:11:04.042110 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.042082    4823 scope.go:122] "RemoveContainer" containerID="d87e9b6a7434c574b39820d760b36e4c8a256411b8d8094015e556907f00cce7"
Oct 30 20:11:04.073712 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.073680    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"]
Oct 30 20:11:04.078927 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.078895    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19f2ee-74c7759855-ks8jd"]
Oct 30 20:11:04.246151 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.246120    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"
Oct 30 20:11:04.314814 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.314773    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dfba44f-22b1-4336-8938-b29d8478811f" path="/var/lib/kubelet/pods/9dfba44f-22b1-4336-8938-b29d8478811f/volumes"
Oct 30 20:11:04.336274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.336245    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26e56c15-b87d-4b0a-b403-f32809781e5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26e56c15-b87d-4b0a-b403-f32809781e5d-maas-api-openshift-ca\") pod \"26e56c15-b87d-4b0a-b403-f32809781e5d\" (UID: \"26e56c15-b87d-4b0a-b403-f32809781e5d\") "
Oct 30 20:11:04.336468 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.336298    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26e56c15-b87d-4b0a-b403-f32809781e5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26e56c15-b87d-4b0a-b403-f32809781e5d-maas-api-xks-ca\") pod \"26e56c15-b87d-4b0a-b403-f32809781e5d\" (UID: \"26e56c15-b87d-4b0a-b403-f32809781e5d\") "
Oct 30 20:11:04.336468 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.336326    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26e56c15-b87d-4b0a-b403-f32809781e5d-kube-api-access-ctdvn\" (UniqueName: \"kubernetes.io/projected/26e56c15-b87d-4b0a-b403-f32809781e5d-kube-api-access-ctdvn\") pod \"26e56c15-b87d-4b0a-b403-f32809781e5d\" (UID: \"26e56c15-b87d-4b0a-b403-f32809781e5d\") "
Oct 30 20:11:04.336667 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.336639    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e56c15-b87d-4b0a-b403-f32809781e5d-maas-api-openshift-ca" pod "26e56c15-b87d-4b0a-b403-f32809781e5d" (UID: "26e56c15-b87d-4b0a-b403-f32809781e5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:04.336966 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.336946    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26e56c15-b87d-4b0a-b403-f32809781e5d-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:04.338615 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.338574    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e56c15-b87d-4b0a-b403-f32809781e5d-maas-api-xks-ca" pod "26e56c15-b87d-4b0a-b403-f32809781e5d" (UID: "26e56c15-b87d-4b0a-b403-f32809781e5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:04.338809 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.338785    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e56c15-b87d-4b0a-b403-f32809781e5d-kube-api-access-ctdvn" pod "26e56c15-b87d-4b0a-b403-f32809781e5d" (UID: "26e56c15-b87d-4b0a-b403-f32809781e5d"). InnerVolumeSpecName "kube-api-access-ctdvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:04.438675 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.438592    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26e56c15-b87d-4b0a-b403-f32809781e5d-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:04.438675 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:04.438627    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctdvn\" (UniqueName: \"kubernetes.io/projected/26e56c15-b87d-4b0a-b403-f32809781e5d-kube-api-access-ctdvn\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:05.054238 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:05.054205    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"
Oct 30 20:11:05.054238 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:05.054221    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq" event={"ID":"26e56c15-b87d-4b0a-b403-f32809781e5d","Type":"ContainerDied","Data":"4f75c7bbf82eceee60876a15c19b2b12648b34a4703b7f28d41642e1c10bc5f9"}
Oct 30 20:11:05.054754 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:05.054282    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f75c7bbf82eceee60876a15c19b2b12648b34a4703b7f28d41642e1c10bc5f9"
Oct 30 20:11:05.510947 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:05.507011    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"]
Oct 30 20:11:05.510947 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:05.507270    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf" podUID="2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" containerName="istio-proxy" containerID="cri-o://ddc86e55ca9e85af299f3cebbd5adfd4db241e0fcca24435b0169ac82ac1f7bc" gracePeriod=30
Oct 30 20:11:06.018502 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.018465    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"]
Oct 30 20:11:06.019242 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.019228    4823 cpu_manager.go:414] "containerMap: removing container" podUID="26e56c15-b87d-4b0a-b403-f32809781e5d" containerName="revoke-keys"
Oct 30 20:11:06.019242 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.019243    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26e56c15-b87d-4b0a-b403-f32809781e5d" containerName="revoke-keys"
Oct 30 20:11:06.019332 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.019281    4823 cpu_manager.go:414] "containerMap: removing container" podUID="eef12d45-3357-4f21-9d3e-3d073a82cc3b" containerName="revoke-keys"
Oct 30 20:11:06.019332 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.019286    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eef12d45-3357-4f21-9d3e-3d073a82cc3b" containerName="revoke-keys"
Oct 30 20:11:06.019332 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.019319    4823 cpu_manager.go:414] "containerMap: removing container" podUID="9dfba44f-22b1-4336-8938-b29d8478811f" containerName="maas-api"
Oct 30 20:11:06.019332 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.019326    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dfba44f-22b1-4336-8938-b29d8478811f" containerName="maas-api"
Oct 30 20:11:06.019462 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.019422    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dfba44f-22b1-4336-8938-b29d8478811f" containerName="maas-api"
Oct 30 20:11:06.019462 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.019430    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="26e56c15-b87d-4b0a-b403-f32809781e5d" containerName="revoke-keys"
Oct 30 20:11:06.019462 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.019442    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="eef12d45-3357-4f21-9d3e-3d073a82cc3b" containerName="revoke-keys"
Oct 30 20:11:06.026116 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.026085    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"
Oct 30 20:11:06.033719 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.033685    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"]
Oct 30 20:11:06.036044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.036019    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:11:06.036908 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.036820    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:11:06.129509 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.129475    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1466041a-57ee-4474-94d9-026d6b8fae6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5\" (UID: \"1466041a-57ee-4474-94d9-026d6b8fae6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"
Oct 30 20:11:06.129892 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.129556    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1466041a-57ee-4474-94d9-026d6b8fae6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5\" (UID: \"1466041a-57ee-4474-94d9-026d6b8fae6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"
Oct 30 20:11:06.129892 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.129692    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m595n\" (UniqueName: \"kubernetes.io/projected/1466041a-57ee-4474-94d9-026d6b8fae6c-kube-api-access-m595n\") pod \"maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5\" (UID: \"1466041a-57ee-4474-94d9-026d6b8fae6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"
Oct 30 20:11:06.231624 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.231580    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1466041a-57ee-4474-94d9-026d6b8fae6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5\" (UID: \"1466041a-57ee-4474-94d9-026d6b8fae6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"
Oct 30 20:11:06.231826 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.231673    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1466041a-57ee-4474-94d9-026d6b8fae6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5\" (UID: \"1466041a-57ee-4474-94d9-026d6b8fae6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"
Oct 30 20:11:06.231826 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.231759    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m595n\" (UniqueName: \"kubernetes.io/projected/1466041a-57ee-4474-94d9-026d6b8fae6c-kube-api-access-m595n\") pod \"maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5\" (UID: \"1466041a-57ee-4474-94d9-026d6b8fae6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"
Oct 30 20:11:06.232440 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.232415    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1466041a-57ee-4474-94d9-026d6b8fae6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5\" (UID: \"1466041a-57ee-4474-94d9-026d6b8fae6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"
Oct 30 20:11:06.234216 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.234194    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1466041a-57ee-4474-94d9-026d6b8fae6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5\" (UID: \"1466041a-57ee-4474-94d9-026d6b8fae6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"
Oct 30 20:11:06.244604 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.244577    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m595n\" (UniqueName: \"kubernetes.io/projected/1466041a-57ee-4474-94d9-026d6b8fae6c-kube-api-access-m595n\") pod \"maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5\" (UID: \"1466041a-57ee-4474-94d9-026d6b8fae6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"
Oct 30 20:11:06.347244 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.347164    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"
Oct 30 20:11:06.683250 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:06.683206    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"]
Oct 30 20:11:06.688205 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:11:06.688171    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1466041a_57ee_4474_94d9_026d6b8fae6c.slice/crio-72f9aca18b913141843c22f231b72a6e12cf1ec03ccec84ddb2fb20c4f08e8da WatchSource:0}: Error finding container 72f9aca18b913141843c22f231b72a6e12cf1ec03ccec84ddb2fb20c4f08e8da: Status 404 returned error can't find the container with id 72f9aca18b913141843c22f231b72a6e12cf1ec03ccec84ddb2fb20c4f08e8da
Oct 30 20:11:07.100742 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:07.100652    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5" event={"ID":"1466041a-57ee-4474-94d9-026d6b8fae6c","Type":"ContainerStarted","Data":"72f9aca18b913141843c22f231b72a6e12cf1ec03ccec84ddb2fb20c4f08e8da"}
Oct 30 20:11:08.044061 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:08.044018    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"]
Oct 30 20:11:08.044602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:08.044277    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d" podUID="9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" containerName="istio-proxy" containerID="cri-o://a638832017626a6a1926db39045c72c992f5b56f42dc5873123116bdbccd2d59" gracePeriod=30
Oct 30 20:11:08.113275 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:08.113234    4823 generic.go:356] "Generic (PLEG): container finished" podID="1466041a-57ee-4474-94d9-026d6b8fae6c" containerID="f44290dcf547bb659391719ef4ef0cb33cfd89e4dc7b54ff8c85d5249b0bf4e7" exitCode=0
Oct 30 20:11:08.113456 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:08.113293    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5" event={"ID":"1466041a-57ee-4474-94d9-026d6b8fae6c","Type":"ContainerDied","Data":"f44290dcf547bb659391719ef4ef0cb33cfd89e4dc7b54ff8c85d5249b0bf4e7"}
Oct 30 20:11:08.584064 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:08.584025    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"]
Oct 30 20:11:08.588951 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:08.588928    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"
Oct 30 20:11:08.698942 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:08.698669    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"]
Oct 30 20:11:08.721820 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:08.716368    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a5c978-c96f-44c0-ac3a-828d454fae89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp\" (UID: \"45a5c978-c96f-44c0-ac3a-828d454fae89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"
Oct 30 20:11:08.721820 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:08.716412    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a5c978-c96f-44c0-ac3a-828d454fae89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp\" (UID: \"45a5c978-c96f-44c0-ac3a-828d454fae89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"
Oct 30 20:11:08.721820 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:08.716565    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqjvd\" (UniqueName: \"kubernetes.io/projected/45a5c978-c96f-44c0-ac3a-828d454fae89-kube-api-access-zqjvd\") pod \"maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp\" (UID: \"45a5c978-c96f-44c0-ac3a-828d454fae89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"
Oct 30 20:11:08.817517 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:08.817481    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a5c978-c96f-44c0-ac3a-828d454fae89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp\" (UID: \"45a5c978-c96f-44c0-ac3a-828d454fae89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"
Oct 30 20:11:08.817717 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:08.817530    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a5c978-c96f-44c0-ac3a-828d454fae89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp\" (UID: \"45a5c978-c96f-44c0-ac3a-828d454fae89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"
Oct 30 20:11:08.817790 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:08.817756    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqjvd\" (UniqueName: \"kubernetes.io/projected/45a5c978-c96f-44c0-ac3a-828d454fae89-kube-api-access-zqjvd\") pod \"maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp\" (UID: \"45a5c978-c96f-44c0-ac3a-828d454fae89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"
Oct 30 20:11:08.818331 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:08.818305    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a5c978-c96f-44c0-ac3a-828d454fae89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp\" (UID: \"45a5c978-c96f-44c0-ac3a-828d454fae89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"
Oct 30 20:11:08.820080 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:08.820044    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a5c978-c96f-44c0-ac3a-828d454fae89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp\" (UID: \"45a5c978-c96f-44c0-ac3a-828d454fae89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"
Oct 30 20:11:08.830431 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:08.830405    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqjvd\" (UniqueName: \"kubernetes.io/projected/45a5c978-c96f-44c0-ac3a-828d454fae89-kube-api-access-zqjvd\") pod \"maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp\" (UID: \"45a5c978-c96f-44c0-ac3a-828d454fae89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"
Oct 30 20:11:09.024974 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:09.024938    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"
Oct 30 20:11:09.347575 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:09.346370    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"]
Oct 30 20:11:09.348887 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:11:09.348820    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a5c978_c96f_44c0_ac3a_828d454fae89.slice/crio-83a5ffa00776157e5db7685ed6efcc59df85fca0dbe51453b37283b13ad7c987 WatchSource:0}: Error finding container 83a5ffa00776157e5db7685ed6efcc59df85fca0dbe51453b37283b13ad7c987: Status 404 returned error can't find the container with id 83a5ffa00776157e5db7685ed6efcc59df85fca0dbe51453b37283b13ad7c987
Oct 30 20:11:09.471894 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:09.460085    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"
Oct 30 20:11:09.529973 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:09.529942    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1466041a-57ee-4474-94d9-026d6b8fae6c-kube-api-access-m595n\" (UniqueName: \"kubernetes.io/projected/1466041a-57ee-4474-94d9-026d6b8fae6c-kube-api-access-m595n\") pod \"1466041a-57ee-4474-94d9-026d6b8fae6c\" (UID: \"1466041a-57ee-4474-94d9-026d6b8fae6c\") "
Oct 30 20:11:09.530143 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:09.530101    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1466041a-57ee-4474-94d9-026d6b8fae6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1466041a-57ee-4474-94d9-026d6b8fae6c-maas-api-xks-ca\") pod \"1466041a-57ee-4474-94d9-026d6b8fae6c\" (UID: \"1466041a-57ee-4474-94d9-026d6b8fae6c\") "
Oct 30 20:11:09.530207 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:09.530163    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1466041a-57ee-4474-94d9-026d6b8fae6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1466041a-57ee-4474-94d9-026d6b8fae6c-maas-api-openshift-ca\") pod \"1466041a-57ee-4474-94d9-026d6b8fae6c\" (UID: \"1466041a-57ee-4474-94d9-026d6b8fae6c\") "
Oct 30 20:11:09.531091 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:09.530576    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1466041a-57ee-4474-94d9-026d6b8fae6c-maas-api-openshift-ca" pod "1466041a-57ee-4474-94d9-026d6b8fae6c" (UID: "1466041a-57ee-4474-94d9-026d6b8fae6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:09.531091 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:09.531056    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1466041a-57ee-4474-94d9-026d6b8fae6c-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:09.532449 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:09.532416    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1466041a-57ee-4474-94d9-026d6b8fae6c-maas-api-xks-ca" pod "1466041a-57ee-4474-94d9-026d6b8fae6c" (UID: "1466041a-57ee-4474-94d9-026d6b8fae6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:09.532551 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:09.532429    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1466041a-57ee-4474-94d9-026d6b8fae6c-kube-api-access-m595n" pod "1466041a-57ee-4474-94d9-026d6b8fae6c" (UID: "1466041a-57ee-4474-94d9-026d6b8fae6c"). InnerVolumeSpecName "kube-api-access-m595n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:09.632909 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:09.632871    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m595n\" (UniqueName: \"kubernetes.io/projected/1466041a-57ee-4474-94d9-026d6b8fae6c-kube-api-access-m595n\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:09.632909 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:09.632903    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1466041a-57ee-4474-94d9-026d6b8fae6c-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:10.151274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.151177    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp" event={"ID":"45a5c978-c96f-44c0-ac3a-828d454fae89","Type":"ContainerStarted","Data":"0f78691a19cc1fc287c341140b35ad70d1500105e3ebb2ff77aa90623f33c023"}
Oct 30 20:11:10.151274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.151235    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp" event={"ID":"45a5c978-c96f-44c0-ac3a-828d454fae89","Type":"ContainerStarted","Data":"83a5ffa00776157e5db7685ed6efcc59df85fca0dbe51453b37283b13ad7c987"}
Oct 30 20:11:10.173886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.153054    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5" event={"ID":"1466041a-57ee-4474-94d9-026d6b8fae6c","Type":"ContainerDied","Data":"72f9aca18b913141843c22f231b72a6e12cf1ec03ccec84ddb2fb20c4f08e8da"}
Oct 30 20:11:10.173886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.153110    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72f9aca18b913141843c22f231b72a6e12cf1ec03ccec84ddb2fb20c4f08e8da"
Oct 30 20:11:10.173886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.153176    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"
Oct 30 20:11:10.194303 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.189246    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp" podStartSLOduration=2.18922099 podStartE2EDuration="2.18922099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:10.183739233 +0000 UTC m=+2470.076353559" watchObservedRunningTime="2026-10-30 20:11:10.18922099 +0000 UTC m=+2470.081835316"
Oct 30 20:11:10.900836 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.900800    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:11:10.998443 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.998411    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-token\" (UniqueName: \"kubernetes.io/projected/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-token\") pod \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") "
Oct 30 20:11:10.998614 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.998583    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-kube-api-access-jqgd9\" (UniqueName: \"kubernetes.io/projected/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-kube-api-access-jqgd9\") pod \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") "
Oct 30 20:11:10.998704 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.998679    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-workload-socket\") pod \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") "
Oct 30 20:11:10.998840 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.998822    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-podinfo\") pod \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") "
Oct 30 20:11:11.000736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.998876    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-workload-certs\") pod \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") "
Oct 30 20:11:11.000736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.998936    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-envoy\") pod \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") "
Oct 30 20:11:11.000736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.998989    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-data\") pod \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") "
Oct 30 20:11:11.000736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.999022    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istiod-ca-cert\") pod \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") "
Oct 30 20:11:11.000736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.999072    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-workload-socket" pod "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" (UID: "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:11.000736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.999099    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-credential-socket\") pod \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\" (UID: \"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3\") "
Oct 30 20:11:11.000736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.999230    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-workload-certs" pod "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" (UID: "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:11.000736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.999287    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-credential-socket" pod "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" (UID: "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:11.000736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.999535    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-data" pod "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" (UID: "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:11.000736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:10.999886    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istiod-ca-cert" pod "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" (UID: "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:11.000736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.000008    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.000736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.000026    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.000736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.000035    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.000736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.000045    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.000736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.000059    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.001949 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.001924    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-podinfo" pod "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" (UID: "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:11:11.002052 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.001988    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-envoy" pod "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" (UID: "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:11.002187 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.002164    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-kube-api-access-jqgd9" pod "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" (UID: "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3"). InnerVolumeSpecName "kube-api-access-jqgd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:11.002510 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.002492    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-token" pod "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" (UID: "2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:11.101995 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.101906    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.101995 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.101933    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.101995 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.101943    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.101995 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.101951    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqgd9\" (UniqueName: \"kubernetes.io/projected/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3-kube-api-access-jqgd9\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:11.193897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.192939    4823 generic.go:356] "Generic (PLEG): container finished" podID="2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" containerID="ddc86e55ca9e85af299f3cebbd5adfd4db241e0fcca24435b0169ac82ac1f7bc" exitCode=0
Oct 30 20:11:11.193897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.193372    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf" event={"ID":"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3","Type":"ContainerDied","Data":"ddc86e55ca9e85af299f3cebbd5adfd4db241e0fcca24435b0169ac82ac1f7bc"}
Oct 30 20:11:11.193897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.193434    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf" event={"ID":"2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3","Type":"ContainerDied","Data":"f6c2aed8505c97870eb77792193111f839907344b2f575d2ba0bf133d3e7811b"}
Oct 30 20:11:11.193897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.193466    4823 scope.go:122] "RemoveContainer" containerID="ddc86e55ca9e85af299f3cebbd5adfd4db241e0fcca24435b0169ac82ac1f7bc"
Oct 30 20:11:11.193897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.193638    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"
Oct 30 20:11:11.216624 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.216486    4823 generic.go:356] "Generic (PLEG): container finished" podID="45a5c978-c96f-44c0-ac3a-828d454fae89" containerID="0f78691a19cc1fc287c341140b35ad70d1500105e3ebb2ff77aa90623f33c023" exitCode=0
Oct 30 20:11:11.219058 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.219011    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp" event={"ID":"45a5c978-c96f-44c0-ac3a-828d454fae89","Type":"ContainerDied","Data":"0f78691a19cc1fc287c341140b35ad70d1500105e3ebb2ff77aa90623f33c023"}
Oct 30 20:11:11.230817 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.230791    4823 scope.go:122] "RemoveContainer" containerID="ddc86e55ca9e85af299f3cebbd5adfd4db241e0fcca24435b0169ac82ac1f7bc"
Oct 30 20:11:11.231203 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:11:11.231178    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc86e55ca9e85af299f3cebbd5adfd4db241e0fcca24435b0169ac82ac1f7bc\": container with ID starting with ddc86e55ca9e85af299f3cebbd5adfd4db241e0fcca24435b0169ac82ac1f7bc not found: ID does not exist" containerID="ddc86e55ca9e85af299f3cebbd5adfd4db241e0fcca24435b0169ac82ac1f7bc"
Oct 30 20:11:11.231291 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.231212    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc86e55ca9e85af299f3cebbd5adfd4db241e0fcca24435b0169ac82ac1f7bc"} err="failed to get container status \"ddc86e55ca9e85af299f3cebbd5adfd4db241e0fcca24435b0169ac82ac1f7bc\": rpc error: code = NotFound desc = could not find container \"ddc86e55ca9e85af299f3cebbd5adfd4db241e0fcca24435b0169ac82ac1f7bc\": container with ID starting with ddc86e55ca9e85af299f3cebbd5adfd4db241e0fcca24435b0169ac82ac1f7bc not found: ID does not exist"
Oct 30 20:11:11.240594 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.240566    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"]
Oct 30 20:11:11.259942 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:11.259824    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-dszzf"]
Oct 30 20:11:12.293655 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:12.293619    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" path="/var/lib/kubelet/pods/2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3/volumes"
Oct 30 20:11:12.758886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:12.758451    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"
Oct 30 20:11:12.844512 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:12.844481    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a5c978-c96f-44c0-ac3a-828d454fae89-kube-api-access-zqjvd\" (UniqueName: \"kubernetes.io/projected/45a5c978-c96f-44c0-ac3a-828d454fae89-kube-api-access-zqjvd\") pod \"45a5c978-c96f-44c0-ac3a-828d454fae89\" (UID: \"45a5c978-c96f-44c0-ac3a-828d454fae89\") "
Oct 30 20:11:12.844728 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:12.844642    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a5c978-c96f-44c0-ac3a-828d454fae89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a5c978-c96f-44c0-ac3a-828d454fae89-maas-api-openshift-ca\") pod \"45a5c978-c96f-44c0-ac3a-828d454fae89\" (UID: \"45a5c978-c96f-44c0-ac3a-828d454fae89\") "
Oct 30 20:11:12.844909 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:12.844856    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45a5c978-c96f-44c0-ac3a-828d454fae89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a5c978-c96f-44c0-ac3a-828d454fae89-maas-api-xks-ca\") pod \"45a5c978-c96f-44c0-ac3a-828d454fae89\" (UID: \"45a5c978-c96f-44c0-ac3a-828d454fae89\") "
Oct 30 20:11:12.845094 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:12.845070    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a5c978-c96f-44c0-ac3a-828d454fae89-maas-api-openshift-ca" pod "45a5c978-c96f-44c0-ac3a-828d454fae89" (UID: "45a5c978-c96f-44c0-ac3a-828d454fae89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:12.845520 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:12.845499    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a5c978-c96f-44c0-ac3a-828d454fae89-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:12.847223 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:12.847193    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45a5c978-c96f-44c0-ac3a-828d454fae89-maas-api-xks-ca" pod "45a5c978-c96f-44c0-ac3a-828d454fae89" (UID: "45a5c978-c96f-44c0-ac3a-828d454fae89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:12.847355 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:12.847238    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a5c978-c96f-44c0-ac3a-828d454fae89-kube-api-access-zqjvd" pod "45a5c978-c96f-44c0-ac3a-828d454fae89" (UID: "45a5c978-c96f-44c0-ac3a-828d454fae89"). InnerVolumeSpecName "kube-api-access-zqjvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:12.946937 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:12.946843    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a5c978-c96f-44c0-ac3a-828d454fae89-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:12.946937 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:12.946888    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqjvd\" (UniqueName: \"kubernetes.io/projected/45a5c978-c96f-44c0-ac3a-828d454fae89-kube-api-access-zqjvd\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:13.295278 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.295225    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp" event={"ID":"45a5c978-c96f-44c0-ac3a-828d454fae89","Type":"ContainerDied","Data":"83a5ffa00776157e5db7685ed6efcc59df85fca0dbe51453b37283b13ad7c987"}
Oct 30 20:11:13.295746 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.295297    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83a5ffa00776157e5db7685ed6efcc59df85fca0dbe51453b37283b13ad7c987"
Oct 30 20:11:13.295746 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.295374    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"
Oct 30 20:11:13.306544 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.306505    4823 generic.go:356] "Generic (PLEG): container finished" podID="9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" containerID="a638832017626a6a1926db39045c72c992f5b56f42dc5873123116bdbccd2d59" exitCode=0
Oct 30 20:11:13.306716 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.306606    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d" event={"ID":"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15","Type":"ContainerDied","Data":"a638832017626a6a1926db39045c72c992f5b56f42dc5873123116bdbccd2d59"}
Oct 30 20:11:13.306716 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.306665    4823 scope.go:122] "RemoveContainer" containerID="7888a3364ad8d80ae22a582ee90f584d2243846afbf27550d4277f74b76ed584"
Oct 30 20:11:13.356924 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.356892    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:11:13.458565 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.458526    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-data\") pod \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") "
Oct 30 20:11:13.458780 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.458751    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-podinfo\") pod \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") "
Oct 30 20:11:13.458942 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.458832    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-data" pod "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" (UID: "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:13.458942 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.458850    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-credential-socket\") pod \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") "
Oct 30 20:11:13.459087 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.458947    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-envoy\") pod \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") "
Oct 30 20:11:13.459087 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.459004    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istiod-ca-cert\") pod \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") "
Oct 30 20:11:13.459087 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.459053    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-kube-api-access-2mdtf\" (UniqueName: \"kubernetes.io/projected/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-kube-api-access-2mdtf\") pod \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") "
Oct 30 20:11:13.459087 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.459081    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-credential-socket" pod "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" (UID: "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:13.459327 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.459305    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-workload-certs\") pod \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") "
Oct 30 20:11:13.459387 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.459355    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-workload-socket\") pod \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") "
Oct 30 20:11:13.459448 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.459401    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istiod-ca-cert" pod "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" (UID: "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:13.459671 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.459652    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-token\" (UniqueName: \"kubernetes.io/projected/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-token\") pod \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\" (UID: \"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15\") "
Oct 30 20:11:13.459765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.459705    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-workload-certs" pod "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" (UID: "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:13.460110 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.460084    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-workload-socket" pod "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" (UID: "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:13.460549 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.460527    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:13.460656 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.460549    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:13.460656 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.460563    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:13.460656 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.460574    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:13.460656 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.460586    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:13.461732 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.461705    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-podinfo" pod "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" (UID: "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:11:13.461732 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.461726    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-kube-api-access-2mdtf" pod "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" (UID: "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15"). InnerVolumeSpecName "kube-api-access-2mdtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:13.462640 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.462620    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-token" pod "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" (UID: "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:13.462973 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.462943    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-envoy" pod "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" (UID: "9aaa5dc0-1ff0-4458-8181-9201b8dc5b15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:13.562876 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.562771    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:13.562876 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.562810    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:13.562876 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.562824    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:13.562876 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.562837    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mdtf\" (UniqueName: \"kubernetes.io/projected/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15-kube-api-access-2mdtf\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:13.725182 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:13.725148    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:11:14.317557 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:14.317514    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d" event={"ID":"9aaa5dc0-1ff0-4458-8181-9201b8dc5b15","Type":"ContainerDied","Data":"b3bc9b939aec6cf61d331f64c325247570a012ba346f3e6b30e25850f5bb2ebc"}
Oct 30 20:11:14.317980 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:14.317595    4823 scope.go:122] "RemoveContainer" containerID="a638832017626a6a1926db39045c72c992f5b56f42dc5873123116bdbccd2d59"
Oct 30 20:11:14.317980 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:14.317733    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"
Oct 30 20:11:14.356876 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:14.356835    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"]
Oct 30 20:11:14.361378 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:14.361351    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-5dc9d"]
Oct 30 20:11:16.282670 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:16.282633    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" path="/var/lib/kubelet/pods/9aaa5dc0-1ff0-4458-8181-9201b8dc5b15/volumes"
Oct 30 20:11:16.884329 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:16.884292    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"]
Oct 30 20:11:16.884551 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:16.884527    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw" podUID="3eb29b61-5760-4a85-8573-a3205e66e833" containerName="istio-proxy" containerID="cri-o://cf51afe5888ea55d599df5293174186b3f670f69e2db99b0127ea60cdfd50c6c" gracePeriod=30
Oct 30 20:11:17.315336 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.315256    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"]
Oct 30 20:11:17.316047 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.316031    4823 cpu_manager.go:414] "containerMap: removing container" podUID="45a5c978-c96f-44c0-ac3a-828d454fae89" containerName="revoke-keys"
Oct 30 20:11:17.316139 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.316048    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a5c978-c96f-44c0-ac3a-828d454fae89" containerName="revoke-keys"
Oct 30 20:11:17.316139 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.316063    4823 cpu_manager.go:414] "containerMap: removing container" podUID="9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" containerName="istio-proxy"
Oct 30 20:11:17.316139 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.316068    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" containerName="istio-proxy"
Oct 30 20:11:17.316139 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.316094    4823 cpu_manager.go:414] "containerMap: removing container" podUID="9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" containerName="istio-proxy"
Oct 30 20:11:17.316139 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.316099    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" containerName="istio-proxy"
Oct 30 20:11:17.316139 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.316121    4823 cpu_manager.go:414] "containerMap: removing container" podUID="1466041a-57ee-4474-94d9-026d6b8fae6c" containerName="revoke-keys"
Oct 30 20:11:17.316139 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.316125    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1466041a-57ee-4474-94d9-026d6b8fae6c" containerName="revoke-keys"
Oct 30 20:11:17.316139 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.316138    4823 cpu_manager.go:414] "containerMap: removing container" podUID="2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" containerName="istio-proxy"
Oct 30 20:11:17.316139 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.316143    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" containerName="istio-proxy"
Oct 30 20:11:17.316410 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.316235    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" containerName="istio-proxy"
Oct 30 20:11:17.316410 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.316246    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="1466041a-57ee-4474-94d9-026d6b8fae6c" containerName="revoke-keys"
Oct 30 20:11:17.316410 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.316254    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e0d0374-f31f-42dc-a1f7-9fc3a4d845e3" containerName="istio-proxy"
Oct 30 20:11:17.316410 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.316262    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a5c978-c96f-44c0-ac3a-828d454fae89" containerName="revoke-keys"
Oct 30 20:11:17.316410 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.316269    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aaa5dc0-1ff0-4458-8181-9201b8dc5b15" containerName="istio-proxy"
Oct 30 20:11:17.336847 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.336821    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.344734 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.344684    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"]
Oct 30 20:11:17.379579 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.379537    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfjxn\" (UniqueName: \"kubernetes.io/projected/b569370e-5fdb-4809-92f1-fcb85b98b9f7-kube-api-access-qfjxn\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.386386 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.379638    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-workload-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.386386 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.379712    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istiod-ca-cert\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.386386 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.379944    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-podinfo\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.386386 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.380073    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-envoy\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.386386 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.380119    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-workload-certs\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.386386 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.380149    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-token\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.386386 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.380225    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-data\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.386386 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.380292    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-credential-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.476720 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.476685    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": read tcp 10.132.0.2:33628->10.132.0.188:15021: read: connection reset by peer" start-of-body=
Oct 30 20:11:17.476842 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.476746    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": read tcp 10.132.0.2:33628->10.132.0.188:15021: read: connection reset by peer"
Oct 30 20:11:17.481561 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.481534    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-podinfo\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.481673 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.481594    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-envoy\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.481673 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.481629    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-workload-certs\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.481673 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.481654    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-token\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.481847 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.481679    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-data\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.481847 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.481700    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-credential-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.482041 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.481949    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfjxn\" (UniqueName: \"kubernetes.io/projected/b569370e-5fdb-4809-92f1-fcb85b98b9f7-kube-api-access-qfjxn\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.482115 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.482091    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-workload-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.482183 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.482116    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-workload-certs\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.482183 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.482162    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istiod-ca-cert\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.482305 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.482253    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-credential-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.482564 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.482538    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-workload-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.483018 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.482991    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-data\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.483157 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.483061    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istiod-ca-cert\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.484747 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.484726    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-podinfo\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.485287 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.485261    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-envoy\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.497210 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.497188    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-token\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.498314 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.498283    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfjxn\" (UniqueName: \"kubernetes.io/projected/b569370e-5fdb-4809-92f1-fcb85b98b9f7-kube-api-access-qfjxn\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.658230 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.658122    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:17.964000 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.963970    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"]
Oct 30 20:11:17.964146 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:11:17.964076    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb569370e_5fdb_4809_92f1_fcb85b98b9f7.slice/crio-a5fa0f72a70b8f2fa585a0240cbd64ff2da689258165d210acd13dda87f73952 WatchSource:0}: Error finding container a5fa0f72a70b8f2fa585a0240cbd64ff2da689258165d210acd13dda87f73952: Status 404 returned error can't find the container with id a5fa0f72a70b8f2fa585a0240cbd64ff2da689258165d210acd13dda87f73952
Oct 30 20:11:17.967023 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.966987    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:11:17.967121 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.967062    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:11:17.967121 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:17.967093    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:11:18.383235 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:18.383199    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7fhfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:11:18.383689 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:18.383252    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw" podUID="3eb29b61-5760-4a85-8573-a3205e66e833" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:11:18.400621 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:18.400579    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l" event={"ID":"b569370e-5fdb-4809-92f1-fcb85b98b9f7","Type":"ContainerStarted","Data":"5fbbfc5ebd47fc01331fb42d8e2c58790dd0b64bc844282cbacaf0da97be4376"}
Oct 30 20:11:18.400798 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:18.400654    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l" event={"ID":"b569370e-5fdb-4809-92f1-fcb85b98b9f7","Type":"ContainerStarted","Data":"a5fa0f72a70b8f2fa585a0240cbd64ff2da689258165d210acd13dda87f73952"}
Oct 30 20:11:18.402523 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:18.402498    4823 generic.go:356] "Generic (PLEG): container finished" podID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerID="85488a16a7ebe75413316d628210bcd7c66c7c19885aa408a882c9732f4f3740" exitCode=0
Oct 30 20:11:18.402640 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:18.402546    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" event={"ID":"98c1ea21-d79b-457c-95f7-45fd2b2befd7","Type":"ContainerDied","Data":"85488a16a7ebe75413316d628210bcd7c66c7c19885aa408a882c9732f4f3740"}
Oct 30 20:11:18.402891 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:18.402857    4823 scope.go:122] "RemoveContainer" containerID="85488a16a7ebe75413316d628210bcd7c66c7c19885aa408a882c9732f4f3740"
Oct 30 20:11:18.407915 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:18.405172    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:11:18.407915 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:18.405342    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:11:18.407915 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:18.405438    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:11:18.442426 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:18.442366    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l" podStartSLOduration=1.44235136 podStartE2EDuration="1.44235136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:18.438830908 +0000 UTC m=+2478.331445231" watchObservedRunningTime="2026-10-30 20:11:18.44235136 +0000 UTC m=+2478.334965749"
Oct 30 20:11:18.661091 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:18.660992    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:19.506879 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:19.504266    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" event={"ID":"98c1ea21-d79b-457c-95f7-45fd2b2befd7","Type":"ContainerStarted","Data":"53e9419693787ca9cad508cf5d087c5af61bba27af43a9da65e78f6223821d09"}
Oct 30 20:11:19.673937 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:19.670931    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 20:11:19.673937 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:19.671011    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l" podUID="b569370e-5fdb-4809-92f1-fcb85b98b9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 20:11:19.698873 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:19.698825    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:11:19.699526 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:19.699501    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:11:19.699740 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:19.699716    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:11:20.423705 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:20.423674    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"]
Oct 30 20:11:20.423922 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:20.423853    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd" podUID="b00c23a6-46e6-4db7-83c0-f8063c327df4" containerName="maas-api" containerID="cri-o://28fc5e0ce7f5b4068c57377001379f15d25303cceb84b2be8dec5b14d8e6595f" gracePeriod=30
Oct 30 20:11:20.663892 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:20.661429    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 20:11:20.663892 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:20.661506    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l" podUID="b569370e-5fdb-4809-92f1-fcb85b98b9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 20:11:20.718942 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:20.718493    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:11:20.718942 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:20.718553    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:11:21.514501 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.514468    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:11:21.551163 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.550490    4823 generic.go:356] "Generic (PLEG): container finished" podID="b00c23a6-46e6-4db7-83c0-f8063c327df4" containerID="28fc5e0ce7f5b4068c57377001379f15d25303cceb84b2be8dec5b14d8e6595f" exitCode=0
Oct 30 20:11:21.551163 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.550595    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"
Oct 30 20:11:21.551163 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.550744    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd" event={"ID":"b00c23a6-46e6-4db7-83c0-f8063c327df4","Type":"ContainerDied","Data":"28fc5e0ce7f5b4068c57377001379f15d25303cceb84b2be8dec5b14d8e6595f"}
Oct 30 20:11:21.551163 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.550794    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd" event={"ID":"b00c23a6-46e6-4db7-83c0-f8063c327df4","Type":"ContainerDied","Data":"74e9901131bb02b8f8c8b6f2e2a72635ec3689213a06961b5c5e5eaf5f88f7c7"}
Oct 30 20:11:21.551163 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.550825    4823 scope.go:122] "RemoveContainer" containerID="28fc5e0ce7f5b4068c57377001379f15d25303cceb84b2be8dec5b14d8e6595f"
Oct 30 20:11:21.622161 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.622129    4823 scope.go:122] "RemoveContainer" containerID="28fc5e0ce7f5b4068c57377001379f15d25303cceb84b2be8dec5b14d8e6595f"
Oct 30 20:11:21.622516 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:11:21.622490    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fc5e0ce7f5b4068c57377001379f15d25303cceb84b2be8dec5b14d8e6595f\": container with ID starting with 28fc5e0ce7f5b4068c57377001379f15d25303cceb84b2be8dec5b14d8e6595f not found: ID does not exist" containerID="28fc5e0ce7f5b4068c57377001379f15d25303cceb84b2be8dec5b14d8e6595f"
Oct 30 20:11:21.622626 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.622524    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fc5e0ce7f5b4068c57377001379f15d25303cceb84b2be8dec5b14d8e6595f"} err="failed to get container status \"28fc5e0ce7f5b4068c57377001379f15d25303cceb84b2be8dec5b14d8e6595f\": rpc error: code = NotFound desc = could not find container \"28fc5e0ce7f5b4068c57377001379f15d25303cceb84b2be8dec5b14d8e6595f\": container with ID starting with 28fc5e0ce7f5b4068c57377001379f15d25303cceb84b2be8dec5b14d8e6595f not found: ID does not exist"
Oct 30 20:11:21.677903 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.671141    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 20:11:21.677903 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.671202    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l" podUID="b569370e-5fdb-4809-92f1-fcb85b98b9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 20:11:21.724612 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.717093    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b00c23a6-46e6-4db7-83c0-f8063c327df4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00c23a6-46e6-4db7-83c0-f8063c327df4-openshift-service-ca\") pod \"b00c23a6-46e6-4db7-83c0-f8063c327df4\" (UID: \"b00c23a6-46e6-4db7-83c0-f8063c327df4\") "
Oct 30 20:11:21.724612 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.717163    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b00c23a6-46e6-4db7-83c0-f8063c327df4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b00c23a6-46e6-4db7-83c0-f8063c327df4-maas-api-tls\") pod \"b00c23a6-46e6-4db7-83c0-f8063c327df4\" (UID: \"b00c23a6-46e6-4db7-83c0-f8063c327df4\") "
Oct 30 20:11:21.724612 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.717354    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00c23a6-46e6-4db7-83c0-f8063c327df4-kube-api-access-5dzbj\" (UniqueName: \"kubernetes.io/projected/b00c23a6-46e6-4db7-83c0-f8063c327df4-kube-api-access-5dzbj\") pod \"b00c23a6-46e6-4db7-83c0-f8063c327df4\" (UID: \"b00c23a6-46e6-4db7-83c0-f8063c327df4\") "
Oct 30 20:11:21.724612 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.717591    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b00c23a6-46e6-4db7-83c0-f8063c327df4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b00c23a6-46e6-4db7-83c0-f8063c327df4-metrics-certs\") pod \"b00c23a6-46e6-4db7-83c0-f8063c327df4\" (UID: \"b00c23a6-46e6-4db7-83c0-f8063c327df4\") "
Oct 30 20:11:21.726936 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.725546    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00c23a6-46e6-4db7-83c0-f8063c327df4-openshift-service-ca" pod "b00c23a6-46e6-4db7-83c0-f8063c327df4" (UID: "b00c23a6-46e6-4db7-83c0-f8063c327df4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:21.727100 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.726988    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:11:21.727100 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.727039    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:11:21.727429 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.727395    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00c23a6-46e6-4db7-83c0-f8063c327df4-kube-api-access-5dzbj" pod "b00c23a6-46e6-4db7-83c0-f8063c327df4" (UID: "b00c23a6-46e6-4db7-83c0-f8063c327df4"). InnerVolumeSpecName "kube-api-access-5dzbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:21.732229 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.732201    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b00c23a6-46e6-4db7-83c0-f8063c327df4-metrics-certs" pod "b00c23a6-46e6-4db7-83c0-f8063c327df4" (UID: "b00c23a6-46e6-4db7-83c0-f8063c327df4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:21.739939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.739895    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b00c23a6-46e6-4db7-83c0-f8063c327df4-maas-api-tls" pod "b00c23a6-46e6-4db7-83c0-f8063c327df4" (UID: "b00c23a6-46e6-4db7-83c0-f8063c327df4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:21.822886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.820039    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dzbj\" (UniqueName: \"kubernetes.io/projected/b00c23a6-46e6-4db7-83c0-f8063c327df4-kube-api-access-5dzbj\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:21.822886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.820112    4823 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b00c23a6-46e6-4db7-83c0-f8063c327df4-metrics-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:21.822886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.820135    4823 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00c23a6-46e6-4db7-83c0-f8063c327df4-openshift-service-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:21.822886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:21.820151    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b00c23a6-46e6-4db7-83c0-f8063c327df4-maas-api-tls\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:22.037849 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.037811    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"]
Oct 30 20:11:22.048072 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.048043    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6986a4-c9b8dbd8f-qntzd"]
Oct 30 20:11:22.443787 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.433241    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b00c23a6-46e6-4db7-83c0-f8063c327df4" path="/var/lib/kubelet/pods/b00c23a6-46e6-4db7-83c0-f8063c327df4/volumes"
Oct 30 20:11:22.638148 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.638114    4823 generic.go:356] "Generic (PLEG): container finished" podID="3eb29b61-5760-4a85-8573-a3205e66e833" containerID="cf51afe5888ea55d599df5293174186b3f670f69e2db99b0127ea60cdfd50c6c" exitCode=0
Oct 30 20:11:22.638323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.638241    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw" event={"ID":"3eb29b61-5760-4a85-8573-a3205e66e833","Type":"ContainerDied","Data":"cf51afe5888ea55d599df5293174186b3f670f69e2db99b0127ea60cdfd50c6c"}
Oct 30 20:11:22.743279 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.743197    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 20:11:22.743835 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.743805    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l" podUID="b569370e-5fdb-4809-92f1-fcb85b98b9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 20:11:22.744084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.744064    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:11:22.744209 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.744188    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:11:22.840191 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.837345    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 20:11:22.933655 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.933624    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb29b61-5760-4a85-8573-a3205e66e833-kube-api-access-p7d2k\" (UniqueName: \"kubernetes.io/projected/3eb29b61-5760-4a85-8573-a3205e66e833-kube-api-access-p7d2k\") pod \"3eb29b61-5760-4a85-8573-a3205e66e833\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") "
Oct 30 20:11:22.933832 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.933755    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-credential-socket\") pod \"3eb29b61-5760-4a85-8573-a3205e66e833\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") "
Oct 30 20:11:22.933832 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.933790    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-workload-socket\") pod \"3eb29b61-5760-4a85-8573-a3205e66e833\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") "
Oct 30 20:11:22.933990 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.933932    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb29b61-5760-4a85-8573-a3205e66e833-istio-token\" (UniqueName: \"kubernetes.io/projected/3eb29b61-5760-4a85-8573-a3205e66e833-istio-token\") pod \"3eb29b61-5760-4a85-8573-a3205e66e833\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") "
Oct 30 20:11:22.934060 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.934037    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-istio-data\") pod \"3eb29b61-5760-4a85-8573-a3205e66e833\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") "
Oct 30 20:11:22.934170 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.934148    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-workload-certs\") pod \"3eb29b61-5760-4a85-8573-a3205e66e833\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") "
Oct 30 20:11:22.934230 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.934205    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb29b61-5760-4a85-8573-a3205e66e833-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb29b61-5760-4a85-8573-a3205e66e833-istiod-ca-cert\") pod \"3eb29b61-5760-4a85-8573-a3205e66e833\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") "
Oct 30 20:11:22.934359 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.934345    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-istio-envoy\") pod \"3eb29b61-5760-4a85-8573-a3205e66e833\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") "
Oct 30 20:11:22.934416 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.934376    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-istio-data" pod "3eb29b61-5760-4a85-8573-a3205e66e833" (UID: "3eb29b61-5760-4a85-8573-a3205e66e833"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:22.934416 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.934389    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3eb29b61-5760-4a85-8573-a3205e66e833-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb29b61-5760-4a85-8573-a3205e66e833-istio-podinfo\") pod \"3eb29b61-5760-4a85-8573-a3205e66e833\" (UID: \"3eb29b61-5760-4a85-8573-a3205e66e833\") "
Oct 30 20:11:22.935333 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.935192    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:22.935448 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.935408    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-credential-socket" pod "3eb29b61-5760-4a85-8573-a3205e66e833" (UID: "3eb29b61-5760-4a85-8573-a3205e66e833"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:22.935571 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.935481    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb29b61-5760-4a85-8573-a3205e66e833-istiod-ca-cert" pod "3eb29b61-5760-4a85-8573-a3205e66e833" (UID: "3eb29b61-5760-4a85-8573-a3205e66e833"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:22.935571 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.935494    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-workload-socket" pod "3eb29b61-5760-4a85-8573-a3205e66e833" (UID: "3eb29b61-5760-4a85-8573-a3205e66e833"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:22.935571 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.935544    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-workload-certs" pod "3eb29b61-5760-4a85-8573-a3205e66e833" (UID: "3eb29b61-5760-4a85-8573-a3205e66e833"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:22.937032 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.937004    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eb29b61-5760-4a85-8573-a3205e66e833-istio-podinfo" pod "3eb29b61-5760-4a85-8573-a3205e66e833" (UID: "3eb29b61-5760-4a85-8573-a3205e66e833"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:11:22.937245 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.937214    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-istio-envoy" pod "3eb29b61-5760-4a85-8573-a3205e66e833" (UID: "3eb29b61-5760-4a85-8573-a3205e66e833"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:22.937323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.937274    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb29b61-5760-4a85-8573-a3205e66e833-istio-token" pod "3eb29b61-5760-4a85-8573-a3205e66e833" (UID: "3eb29b61-5760-4a85-8573-a3205e66e833"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:22.943839 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:22.943781    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb29b61-5760-4a85-8573-a3205e66e833-kube-api-access-p7d2k" pod "3eb29b61-5760-4a85-8573-a3205e66e833" (UID: "3eb29b61-5760-4a85-8573-a3205e66e833"). InnerVolumeSpecName "kube-api-access-p7d2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:23.056198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.053252    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb29b61-5760-4a85-8573-a3205e66e833-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:23.056198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.053296    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:23.056198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.053310    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb29b61-5760-4a85-8573-a3205e66e833-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:23.056198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.053323    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:23.056198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.053336    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb29b61-5760-4a85-8573-a3205e66e833-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:23.056198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.053350    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7d2k\" (UniqueName: \"kubernetes.io/projected/3eb29b61-5760-4a85-8573-a3205e66e833-kube-api-access-p7d2k\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:23.056198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.053366    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:23.056198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.053379    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb29b61-5760-4a85-8573-a3205e66e833-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:23.649657 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.649630    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"
Oct 30 20:11:23.649902 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.649845    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw" event={"ID":"3eb29b61-5760-4a85-8573-a3205e66e833","Type":"ContainerDied","Data":"2614457c65348f8a3b9afce5574af11aa3241dff590be4ac9a2dbe07d32ab22e"}
Oct 30 20:11:23.650026 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.650016    4823 scope.go:122] "RemoveContainer" containerID="cf51afe5888ea55d599df5293174186b3f670f69e2db99b0127ea60cdfd50c6c"
Oct 30 20:11:23.667458 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.667423    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 20:11:23.709193 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.708992    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l" podUID="b569370e-5fdb-4809-92f1-fcb85b98b9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 20:11:23.709278 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.705532    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:11:23.709347 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.709293    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:11:23.732492 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.732454    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"]
Oct 30 20:11:23.736708 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:23.736675    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7fhfw"]
Oct 30 20:11:24.606108 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.606062    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb29b61-5760-4a85-8573-a3205e66e833" path="/var/lib/kubelet/pods/3eb29b61-5760-4a85-8573-a3205e66e833/volumes"
Oct 30 20:11:24.607190 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.607149    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"]
Oct 30 20:11:24.608573 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.608532    4823 cpu_manager.go:414] "containerMap: removing container" podUID="b00c23a6-46e6-4db7-83c0-f8063c327df4" containerName="maas-api"
Oct 30 20:11:24.608744 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.608727    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00c23a6-46e6-4db7-83c0-f8063c327df4" containerName="maas-api"
Oct 30 20:11:24.609113 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.609096    4823 cpu_manager.go:414] "containerMap: removing container" podUID="3eb29b61-5760-4a85-8573-a3205e66e833" containerName="istio-proxy"
Oct 30 20:11:24.609253 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.609239    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb29b61-5760-4a85-8573-a3205e66e833" containerName="istio-proxy"
Oct 30 20:11:24.609561 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.609528    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00c23a6-46e6-4db7-83c0-f8063c327df4" containerName="maas-api"
Oct 30 20:11:24.609709 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.609695    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb29b61-5760-4a85-8573-a3205e66e833" containerName="istio-proxy"
Oct 30 20:11:24.621703 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.621676    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"
Oct 30 20:11:24.627897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.627850    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"]
Oct 30 20:11:24.628737 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.628719    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:11:24.628943 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.628919    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:11:24.680887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.677302    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 20:11:24.680887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.677375    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l" podUID="b569370e-5fdb-4809-92f1-fcb85b98b9f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 20:11:24.701288 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.701247    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:11:24.703997 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.703957    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:11:24.708266 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.707399    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcx4x\" (UniqueName: \"kubernetes.io/projected/39484f68-3e11-417e-b0a7-d562ad75a758-kube-api-access-vcx4x\") pod \"maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k\" (UID: \"39484f68-3e11-417e-b0a7-d562ad75a758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"
Oct 30 20:11:24.708266 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.707647    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39484f68-3e11-417e-b0a7-d562ad75a758-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k\" (UID: \"39484f68-3e11-417e-b0a7-d562ad75a758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"
Oct 30 20:11:24.708266 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.707683    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39484f68-3e11-417e-b0a7-d562ad75a758-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k\" (UID: \"39484f68-3e11-417e-b0a7-d562ad75a758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"
Oct 30 20:11:24.809155 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.809120    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39484f68-3e11-417e-b0a7-d562ad75a758-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k\" (UID: \"39484f68-3e11-417e-b0a7-d562ad75a758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"
Oct 30 20:11:24.809155 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.809170    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39484f68-3e11-417e-b0a7-d562ad75a758-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k\" (UID: \"39484f68-3e11-417e-b0a7-d562ad75a758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"
Oct 30 20:11:24.809412 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.809333    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcx4x\" (UniqueName: \"kubernetes.io/projected/39484f68-3e11-417e-b0a7-d562ad75a758-kube-api-access-vcx4x\") pod \"maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k\" (UID: \"39484f68-3e11-417e-b0a7-d562ad75a758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"
Oct 30 20:11:24.810322 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.810284    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39484f68-3e11-417e-b0a7-d562ad75a758-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k\" (UID: \"39484f68-3e11-417e-b0a7-d562ad75a758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"
Oct 30 20:11:24.812841 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.812815    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39484f68-3e11-417e-b0a7-d562ad75a758-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k\" (UID: \"39484f68-3e11-417e-b0a7-d562ad75a758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"
Oct 30 20:11:24.824672 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.824640    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcx4x\" (UniqueName: \"kubernetes.io/projected/39484f68-3e11-417e-b0a7-d562ad75a758-kube-api-access-vcx4x\") pod \"maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k\" (UID: \"39484f68-3e11-417e-b0a7-d562ad75a758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"
Oct 30 20:11:24.940217 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:24.940127    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"
Oct 30 20:11:25.295694 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:25.295667    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"]
Oct 30 20:11:25.297199 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:11:25.297170    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39484f68_3e11_417e_b0a7_d562ad75a758.slice/crio-4c46a396126cd6259a82b1e97eff00012864ee2dc6f9a7206a8efd07dffb6e41 WatchSource:0}: Error finding container 4c46a396126cd6259a82b1e97eff00012864ee2dc6f9a7206a8efd07dffb6e41: Status 404 returned error can't find the container with id 4c46a396126cd6259a82b1e97eff00012864ee2dc6f9a7206a8efd07dffb6e41
Oct 30 20:11:25.685639 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:25.685336    4823 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 20:11:25.685639 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:25.685394    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 20:11:25.734926 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:25.734353    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:25.735557 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:25.735532    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:25.740918 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:25.740476    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:25.747343 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:25.747299    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k" event={"ID":"39484f68-3e11-417e-b0a7-d562ad75a758","Type":"ContainerStarted","Data":"4c46a396126cd6259a82b1e97eff00012864ee2dc6f9a7206a8efd07dffb6e41"}
Oct 30 20:11:26.689851 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:26.689771    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:11:26.690317 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:26.690302    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:11:26.691003 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:26.690980    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:11:26.759552 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:26.759519    4823 generic.go:356] "Generic (PLEG): container finished" podID="39484f68-3e11-417e-b0a7-d562ad75a758" containerID="de2406bac5ae22ebc9167f54edcc6ed76fb912c611ccacf5474b01e4fa18be6a" exitCode=0
Oct 30 20:11:26.759732 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:26.759612    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k" event={"ID":"39484f68-3e11-417e-b0a7-d562ad75a758","Type":"ContainerDied","Data":"de2406bac5ae22ebc9167f54edcc6ed76fb912c611ccacf5474b01e4fa18be6a"}
Oct 30 20:11:27.953335 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:27.953301    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"
Oct 30 20:11:27.999399 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:27.999351    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39484f68-3e11-417e-b0a7-d562ad75a758-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39484f68-3e11-417e-b0a7-d562ad75a758-maas-api-openshift-ca\") pod \"39484f68-3e11-417e-b0a7-d562ad75a758\" (UID: \"39484f68-3e11-417e-b0a7-d562ad75a758\") "
Oct 30 20:11:27.999613 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:27.999453    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39484f68-3e11-417e-b0a7-d562ad75a758-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39484f68-3e11-417e-b0a7-d562ad75a758-maas-api-xks-ca\") pod \"39484f68-3e11-417e-b0a7-d562ad75a758\" (UID: \"39484f68-3e11-417e-b0a7-d562ad75a758\") "
Oct 30 20:11:27.999613 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:27.999555    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39484f68-3e11-417e-b0a7-d562ad75a758-kube-api-access-vcx4x\" (UniqueName: \"kubernetes.io/projected/39484f68-3e11-417e-b0a7-d562ad75a758-kube-api-access-vcx4x\") pod \"39484f68-3e11-417e-b0a7-d562ad75a758\" (UID: \"39484f68-3e11-417e-b0a7-d562ad75a758\") "
Oct 30 20:11:27.999932 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:27.999903    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39484f68-3e11-417e-b0a7-d562ad75a758-maas-api-openshift-ca" pod "39484f68-3e11-417e-b0a7-d562ad75a758" (UID: "39484f68-3e11-417e-b0a7-d562ad75a758"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:28.000346 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:28.000321    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39484f68-3e11-417e-b0a7-d562ad75a758-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:28.004361 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:28.004330    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39484f68-3e11-417e-b0a7-d562ad75a758-kube-api-access-vcx4x" pod "39484f68-3e11-417e-b0a7-d562ad75a758" (UID: "39484f68-3e11-417e-b0a7-d562ad75a758"). InnerVolumeSpecName "kube-api-access-vcx4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:28.005681 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:28.005653    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39484f68-3e11-417e-b0a7-d562ad75a758-maas-api-xks-ca" pod "39484f68-3e11-417e-b0a7-d562ad75a758" (UID: "39484f68-3e11-417e-b0a7-d562ad75a758"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:28.101738 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:28.101649    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39484f68-3e11-417e-b0a7-d562ad75a758-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:28.101738 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:28.101689    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcx4x\" (UniqueName: \"kubernetes.io/projected/39484f68-3e11-417e-b0a7-d562ad75a758-kube-api-access-vcx4x\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:28.781941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:28.781909    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"
Oct 30 20:11:28.781941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:28.781934    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k" event={"ID":"39484f68-3e11-417e-b0a7-d562ad75a758","Type":"ContainerDied","Data":"4c46a396126cd6259a82b1e97eff00012864ee2dc6f9a7206a8efd07dffb6e41"}
Oct 30 20:11:28.782185 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:28.781982    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c46a396126cd6259a82b1e97eff00012864ee2dc6f9a7206a8efd07dffb6e41"
Oct 30 20:11:28.850205 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:28.850161    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"]
Oct 30 20:11:28.850410 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:28.850379    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy" containerID="cri-o://53e9419693787ca9cad508cf5d087c5af61bba27af43a9da65e78f6223821d09" gracePeriod=30
Oct 30 20:11:28.856334 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:28.856309    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"]
Oct 30 20:11:28.856523 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:28.856501    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l" podUID="b569370e-5fdb-4809-92f1-fcb85b98b9f7" containerName="istio-proxy" containerID="cri-o://5fbbfc5ebd47fc01331fb42d8e2c58790dd0b64bc844282cbacaf0da97be4376" gracePeriod=30
Oct 30 20:11:29.598809 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.598768    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"]
Oct 30 20:11:29.600654 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.600622    4823 cpu_manager.go:414] "containerMap: removing container" podUID="39484f68-3e11-417e-b0a7-d562ad75a758" containerName="revoke-keys"
Oct 30 20:11:29.600654 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.600655    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39484f68-3e11-417e-b0a7-d562ad75a758" containerName="revoke-keys"
Oct 30 20:11:29.601135 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.601113    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="39484f68-3e11-417e-b0a7-d562ad75a758" containerName="revoke-keys"
Oct 30 20:11:29.605586 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.605559    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.607334 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.607305    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"]
Oct 30 20:11:29.610200 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.610164    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-274922\"/\"openshift-service-ca.crt\""
Oct 30 20:11:29.610466 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.610444    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-274922\"/\"kube-root-ca.crt\""
Oct 30 20:11:29.622141 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.622117    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-274922\"/\"default-dockercfg-76g5k\""
Oct 30 20:11:29.622301 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.622202    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-274922\"/\"auth-test-model-274922-kserve-self-signed-certs\""
Oct 30 20:11:29.660131 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.660097    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-tls-certs\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.660293 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.660152    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-tmp-dir\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.660293 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.660223    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwkxn\" (UniqueName: \"kubernetes.io/projected/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-kube-api-access-vwkxn\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.660378 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.660315    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-dshm\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.660432 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.660419    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-home\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.660490 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.660476    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-model-cache\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.678981 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.678942    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"]
Oct 30 20:11:29.683640 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.683617    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"
Oct 30 20:11:29.687430 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.687403    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:11:29.687601 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.687531    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:11:29.691969 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.691940    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"]
Oct 30 20:11:29.762113 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.762080    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d91f3d0-a115-4e98-a683-a8fd017d89d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf\" (UID: \"3d91f3d0-a115-4e98-a683-a8fd017d89d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"
Oct 30 20:11:29.762278 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.762144    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-home\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.762278 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.762245    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-model-cache\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.762387 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.762374    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d91f3d0-a115-4e98-a683-a8fd017d89d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf\" (UID: \"3d91f3d0-a115-4e98-a683-a8fd017d89d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"
Oct 30 20:11:29.762445 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.762415    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-tls-certs\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.762522 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.762500    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-tmp-dir\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.762582 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.762521    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-home\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.762582 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.762542    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-model-cache\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.762582 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.762561    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwkxn\" (UniqueName: \"kubernetes.io/projected/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-kube-api-access-vwkxn\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.762738 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.762636    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssvsq\" (UniqueName: \"kubernetes.io/projected/3d91f3d0-a115-4e98-a683-a8fd017d89d9-kube-api-access-ssvsq\") pod \"maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf\" (UID: \"3d91f3d0-a115-4e98-a683-a8fd017d89d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"
Oct 30 20:11:29.762738 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.762683    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-dshm\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.762851 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.762759    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-tmp-dir\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.764856 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.764833    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-dshm\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.765111 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.765093    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-tls-certs\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.877906 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.877460    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d91f3d0-a115-4e98-a683-a8fd017d89d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf\" (UID: \"3d91f3d0-a115-4e98-a683-a8fd017d89d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"
Oct 30 20:11:29.877906 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.877674    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssvsq\" (UniqueName: \"kubernetes.io/projected/3d91f3d0-a115-4e98-a683-a8fd017d89d9-kube-api-access-ssvsq\") pod \"maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf\" (UID: \"3d91f3d0-a115-4e98-a683-a8fd017d89d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"
Oct 30 20:11:29.877906 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.877815    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d91f3d0-a115-4e98-a683-a8fd017d89d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf\" (UID: \"3d91f3d0-a115-4e98-a683-a8fd017d89d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"
Oct 30 20:11:29.892892 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.892751    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d91f3d0-a115-4e98-a683-a8fd017d89d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf\" (UID: \"3d91f3d0-a115-4e98-a683-a8fd017d89d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"
Oct 30 20:11:29.893088 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.893004    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwkxn\" (UniqueName: \"kubernetes.io/projected/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-kube-api-access-vwkxn\") pod \"auth-test-model-274922-kserve-5c748fcb4f-9bz9h\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") " pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.929960 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.921005    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d91f3d0-a115-4e98-a683-a8fd017d89d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf\" (UID: \"3d91f3d0-a115-4e98-a683-a8fd017d89d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"
Oct 30 20:11:29.929960 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.927001    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:29.964264 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:29.963719    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssvsq\" (UniqueName: \"kubernetes.io/projected/3d91f3d0-a115-4e98-a683-a8fd017d89d9-kube-api-access-ssvsq\") pod \"maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf\" (UID: \"3d91f3d0-a115-4e98-a683-a8fd017d89d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"
Oct 30 20:11:30.007797 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:30.007756    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"
Oct 30 20:11:30.266779 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:30.266747    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"]
Oct 30 20:11:30.267147 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:11:30.267119    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43b70bdc_bb09_4baa_ac3c_dbbefb3adf27.slice/crio-6aa24e9c8ce68d1c0301f9ac2b54b1a20c8a428ca7200b7f5bbc7f1db39ba8f7 WatchSource:0}: Error finding container 6aa24e9c8ce68d1c0301f9ac2b54b1a20c8a428ca7200b7f5bbc7f1db39ba8f7: Status 404 returned error can't find the container with id 6aa24e9c8ce68d1c0301f9ac2b54b1a20c8a428ca7200b7f5bbc7f1db39ba8f7
Oct 30 20:11:30.317518 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:30.317494    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"]
Oct 30 20:11:30.319511 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:11:30.319486    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d91f3d0_a115_4e98_a683_a8fd017d89d9.slice/crio-b5fd422c5b1af2c8b82b8376a42ac9f95a47062ce4c05bdc6ef6519c5105a52c WatchSource:0}: Error finding container b5fd422c5b1af2c8b82b8376a42ac9f95a47062ce4c05bdc6ef6519c5105a52c: Status 404 returned error can't find the container with id b5fd422c5b1af2c8b82b8376a42ac9f95a47062ce4c05bdc6ef6519c5105a52c
Oct 30 20:11:30.929160 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:30.929123    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h" event={"ID":"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27","Type":"ContainerStarted","Data":"daf4d078c05165ed44d91977998dba86f413ded0fd8942be8e3a12e0755f3ce3"}
Oct 30 20:11:30.929564 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:30.929177    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h" event={"ID":"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27","Type":"ContainerStarted","Data":"6aa24e9c8ce68d1c0301f9ac2b54b1a20c8a428ca7200b7f5bbc7f1db39ba8f7"}
Oct 30 20:11:30.930214 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:30.930189    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:30.931247 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:30.931216    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf" event={"ID":"3d91f3d0-a115-4e98-a683-a8fd017d89d9","Type":"ContainerStarted","Data":"435434b095f86254450d0d239e3b672da8bcd0eeeb7aec4129f7b174015f6f29"}
Oct 30 20:11:30.931352 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:30.931277    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf" event={"ID":"3d91f3d0-a115-4e98-a683-a8fd017d89d9","Type":"ContainerStarted","Data":"b5fd422c5b1af2c8b82b8376a42ac9f95a47062ce4c05bdc6ef6519c5105a52c"}
Oct 30 20:11:30.942055 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:30.941811    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"]
Oct 30 20:11:30.945707 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:30.945682    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"
Oct 30 20:11:30.963366 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:30.963316    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"]
Oct 30 20:11:31.004315 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.004247    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h" podStartSLOduration=2.004226535 podStartE2EDuration="2.004226535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:30.981021471 +0000 UTC m=+2490.873635796" watchObservedRunningTime="2026-10-30 20:11:31.004226535 +0000 UTC m=+2490.896840862"
Oct 30 20:11:31.016013 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.015978    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5\" (UID: \"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"
Oct 30 20:11:31.017209 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.017177    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45gkh\" (UniqueName: \"kubernetes.io/projected/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-kube-api-access-45gkh\") pod \"maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5\" (UID: \"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"
Oct 30 20:11:31.017659 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.017292    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5\" (UID: \"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"
Oct 30 20:11:31.018679 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.018639    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf" podStartSLOduration=2.018628094 podStartE2EDuration="2.018628094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:31.018326249 +0000 UTC m=+2490.910940573" watchObservedRunningTime="2026-10-30 20:11:31.018628094 +0000 UTC m=+2490.911242418"
Oct 30 20:11:31.119829 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.119792    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5\" (UID: \"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"
Oct 30 20:11:31.120782 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.120756    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5\" (UID: \"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"
Oct 30 20:11:31.122688 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.122662    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45gkh\" (UniqueName: \"kubernetes.io/projected/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-kube-api-access-45gkh\") pod \"maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5\" (UID: \"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"
Oct 30 20:11:31.122804 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.122784    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5\" (UID: \"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"
Oct 30 20:11:31.125941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.125915    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5\" (UID: \"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"
Oct 30 20:11:31.134494 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.134456    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45gkh\" (UniqueName: \"kubernetes.io/projected/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-kube-api-access-45gkh\") pod \"maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5\" (UID: \"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"
Oct 30 20:11:31.263067 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.263027    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"
Oct 30 20:11:31.551397 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.551366    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"]
Oct 30 20:11:31.552358 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:11:31.552336    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c459648_15c9_4a2f_bb8c_0d7eb3b7ef3f.slice/crio-f5038880280fa15252f8cd6deb7eb0dd05158cf16903e2e0a7d436a13ae29ada WatchSource:0}: Error finding container f5038880280fa15252f8cd6deb7eb0dd05158cf16903e2e0a7d436a13ae29ada: Status 404 returned error can't find the container with id f5038880280fa15252f8cd6deb7eb0dd05158cf16903e2e0a7d436a13ae29ada
Oct 30 20:11:31.942632 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.942597    4823 generic.go:356] "Generic (PLEG): container finished" podID="3d91f3d0-a115-4e98-a683-a8fd017d89d9" containerID="435434b095f86254450d0d239e3b672da8bcd0eeeb7aec4129f7b174015f6f29" exitCode=0
Oct 30 20:11:31.943157 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.942681    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf" event={"ID":"3d91f3d0-a115-4e98-a683-a8fd017d89d9","Type":"ContainerDied","Data":"435434b095f86254450d0d239e3b672da8bcd0eeeb7aec4129f7b174015f6f29"}
Oct 30 20:11:31.944539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.944512    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5" event={"ID":"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f","Type":"ContainerStarted","Data":"f55e49f7b7532e7935ee3ca99aa94180bc95029f3963ac17f883e4b347c4ff24"}
Oct 30 20:11:31.944674 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:31.944556    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5" event={"ID":"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f","Type":"ContainerStarted","Data":"f5038880280fa15252f8cd6deb7eb0dd05158cf16903e2e0a7d436a13ae29ada"}
Oct 30 20:11:32.064998 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:11:32.064898    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:11:32.068722 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:32.068660    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5" podStartSLOduration=2.068640276 podStartE2EDuration="2.068640276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:32.060342973 +0000 UTC m=+2491.952957298" watchObservedRunningTime="2026-10-30 20:11:32.068640276 +0000 UTC m=+2491.961254602"
Oct 30 20:11:32.893091 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:32.893058    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"]
Oct 30 20:11:32.893405 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:32.893291    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk" podUID="96d2936e-4782-4bc3-b4a3-2ebb19701fdc" containerName="maas-api" containerID="cri-o://5ae5f60038a8a56c9d2be2b90dc60ee3951d36ed938fc3c2fe3fd31bc6678343" gracePeriod=30
Oct 30 20:11:32.956470 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:32.956428    4823 generic.go:356] "Generic (PLEG): container finished" podID="2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f" containerID="f55e49f7b7532e7935ee3ca99aa94180bc95029f3963ac17f883e4b347c4ff24" exitCode=0
Oct 30 20:11:32.957023 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:32.956505    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5" event={"ID":"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f","Type":"ContainerDied","Data":"f55e49f7b7532e7935ee3ca99aa94180bc95029f3963ac17f883e4b347c4ff24"}
Oct 30 20:11:33.329048 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.328589    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"
Oct 30 20:11:33.488223 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.475991    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d91f3d0-a115-4e98-a683-a8fd017d89d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d91f3d0-a115-4e98-a683-a8fd017d89d9-maas-api-openshift-ca\") pod \"3d91f3d0-a115-4e98-a683-a8fd017d89d9\" (UID: \"3d91f3d0-a115-4e98-a683-a8fd017d89d9\") "
Oct 30 20:11:33.488223 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.476210    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d91f3d0-a115-4e98-a683-a8fd017d89d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d91f3d0-a115-4e98-a683-a8fd017d89d9-maas-api-xks-ca\") pod \"3d91f3d0-a115-4e98-a683-a8fd017d89d9\" (UID: \"3d91f3d0-a115-4e98-a683-a8fd017d89d9\") "
Oct 30 20:11:33.488223 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.476348    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d91f3d0-a115-4e98-a683-a8fd017d89d9-kube-api-access-ssvsq\" (UniqueName: \"kubernetes.io/projected/3d91f3d0-a115-4e98-a683-a8fd017d89d9-kube-api-access-ssvsq\") pod \"3d91f3d0-a115-4e98-a683-a8fd017d89d9\" (UID: \"3d91f3d0-a115-4e98-a683-a8fd017d89d9\") "
Oct 30 20:11:33.525337 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.525299    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d91f3d0-a115-4e98-a683-a8fd017d89d9-maas-api-openshift-ca" pod "3d91f3d0-a115-4e98-a683-a8fd017d89d9" (UID: "3d91f3d0-a115-4e98-a683-a8fd017d89d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:33.529679 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.527061    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d91f3d0-a115-4e98-a683-a8fd017d89d9-maas-api-xks-ca" pod "3d91f3d0-a115-4e98-a683-a8fd017d89d9" (UID: "3d91f3d0-a115-4e98-a683-a8fd017d89d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:33.581023 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.580505    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d91f3d0-a115-4e98-a683-a8fd017d89d9-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:33.581023 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.580543    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d91f3d0-a115-4e98-a683-a8fd017d89d9-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:33.583121 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.582931    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d91f3d0-a115-4e98-a683-a8fd017d89d9-kube-api-access-ssvsq" pod "3d91f3d0-a115-4e98-a683-a8fd017d89d9" (UID: "3d91f3d0-a115-4e98-a683-a8fd017d89d9"). InnerVolumeSpecName "kube-api-access-ssvsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:33.636667 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.636635    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:11:33.682620 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.682573    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssvsq\" (UniqueName: \"kubernetes.io/projected/3d91f3d0-a115-4e98-a683-a8fd017d89d9-kube-api-access-ssvsq\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:33.779651 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.779569    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"]
Oct 30 20:11:33.783414 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.783383    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-maas-api-tls\") pod \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") "
Oct 30 20:11:33.783608 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.783591    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-metrics-certs\") pod \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") "
Oct 30 20:11:33.783688 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.783674    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-kube-api-access-hlftl\" (UniqueName: \"kubernetes.io/projected/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-kube-api-access-hlftl\") pod \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") "
Oct 30 20:11:33.783809 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.783784    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-openshift-service-ca\") pod \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\" (UID: \"96d2936e-4782-4bc3-b4a3-2ebb19701fdc\") "
Oct 30 20:11:33.784275 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.784226    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-db02a7-t2pw5"]
Oct 30 20:11:33.784275 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.784246    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-openshift-service-ca" pod "96d2936e-4782-4bc3-b4a3-2ebb19701fdc" (UID: "96d2936e-4782-4bc3-b4a3-2ebb19701fdc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:33.784673 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.784649    4823 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-openshift-service-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:33.785750 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.785727    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-maas-api-tls" pod "96d2936e-4782-4bc3-b4a3-2ebb19701fdc" (UID: "96d2936e-4782-4bc3-b4a3-2ebb19701fdc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:33.785911 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.785879    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-metrics-certs" pod "96d2936e-4782-4bc3-b4a3-2ebb19701fdc" (UID: "96d2936e-4782-4bc3-b4a3-2ebb19701fdc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:33.785969 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.785935    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-kube-api-access-hlftl" pod "96d2936e-4782-4bc3-b4a3-2ebb19701fdc" (UID: "96d2936e-4782-4bc3-b4a3-2ebb19701fdc"). InnerVolumeSpecName "kube-api-access-hlftl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:33.898269 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.889010    4823 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-metrics-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:33.898269 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.889048    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlftl\" (UniqueName: \"kubernetes.io/projected/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-kube-api-access-hlftl\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:33.898269 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:33.889062    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96d2936e-4782-4bc3-b4a3-2ebb19701fdc-maas-api-tls\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.006218 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.006114    4823 generic.go:356] "Generic (PLEG): container finished" podID="96d2936e-4782-4bc3-b4a3-2ebb19701fdc" containerID="5ae5f60038a8a56c9d2be2b90dc60ee3951d36ed938fc3c2fe3fd31bc6678343" exitCode=0
Oct 30 20:11:34.006218 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.006195    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"
Oct 30 20:11:34.006642 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.006303    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk" event={"ID":"96d2936e-4782-4bc3-b4a3-2ebb19701fdc","Type":"ContainerDied","Data":"5ae5f60038a8a56c9d2be2b90dc60ee3951d36ed938fc3c2fe3fd31bc6678343"}
Oct 30 20:11:34.006642 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.006356    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk" event={"ID":"96d2936e-4782-4bc3-b4a3-2ebb19701fdc","Type":"ContainerDied","Data":"d78d3a9e1ecdf2b02554ddc50db91eb83592ed574a469b4ff4ad384e63204528"}
Oct 30 20:11:34.006642 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.006380    4823 scope.go:122] "RemoveContainer" containerID="5ae5f60038a8a56c9d2be2b90dc60ee3951d36ed938fc3c2fe3fd31bc6678343"
Oct 30 20:11:34.011815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.011787    4823 generic.go:356] "Generic (PLEG): container finished" podID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerID="53e9419693787ca9cad508cf5d087c5af61bba27af43a9da65e78f6223821d09" exitCode=0
Oct 30 20:11:34.011982 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.011896    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" event={"ID":"98c1ea21-d79b-457c-95f7-45fd2b2befd7","Type":"ContainerDied","Data":"53e9419693787ca9cad508cf5d087c5af61bba27af43a9da65e78f6223821d09"}
Oct 30 20:11:34.013690 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.013664    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf" event={"ID":"3d91f3d0-a115-4e98-a683-a8fd017d89d9","Type":"ContainerDied","Data":"b5fd422c5b1af2c8b82b8376a42ac9f95a47062ce4c05bdc6ef6519c5105a52c"}
Oct 30 20:11:34.013803 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.013701    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"
Oct 30 20:11:34.013803 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.013711    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5fd422c5b1af2c8b82b8376a42ac9f95a47062ce4c05bdc6ef6519c5105a52c"
Oct 30 20:11:34.058191 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.055908    4823 scope.go:122] "RemoveContainer" containerID="5ae5f60038a8a56c9d2be2b90dc60ee3951d36ed938fc3c2fe3fd31bc6678343"
Oct 30 20:11:34.058191 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:11:34.056276    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae5f60038a8a56c9d2be2b90dc60ee3951d36ed938fc3c2fe3fd31bc6678343\": container with ID starting with 5ae5f60038a8a56c9d2be2b90dc60ee3951d36ed938fc3c2fe3fd31bc6678343 not found: ID does not exist" containerID="5ae5f60038a8a56c9d2be2b90dc60ee3951d36ed938fc3c2fe3fd31bc6678343"
Oct 30 20:11:34.058191 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.056316    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae5f60038a8a56c9d2be2b90dc60ee3951d36ed938fc3c2fe3fd31bc6678343"} err="failed to get container status \"5ae5f60038a8a56c9d2be2b90dc60ee3951d36ed938fc3c2fe3fd31bc6678343\": rpc error: code = NotFound desc = could not find container \"5ae5f60038a8a56c9d2be2b90dc60ee3951d36ed938fc3c2fe3fd31bc6678343\": container with ID starting with 5ae5f60038a8a56c9d2be2b90dc60ee3951d36ed938fc3c2fe3fd31bc6678343 not found: ID does not exist"
Oct 30 20:11:34.058191 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.056347    4823 scope.go:122] "RemoveContainer" containerID="85488a16a7ebe75413316d628210bcd7c66c7c19885aa408a882c9732f4f3740"
Oct 30 20:11:34.085006 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.084976    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"]
Oct 30 20:11:34.089641 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.089609    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-db02a7-7b6dbd6c5f-7ckqk"]
Oct 30 20:11:34.248078 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.248052    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:11:34.253090 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.253059    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:34.301539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.301466    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1466041a-57ee-4474-94d9-026d6b8fae6c" path="/var/lib/kubelet/pods/1466041a-57ee-4474-94d9-026d6b8fae6c/volumes"
Oct 30 20:11:34.302124 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.302105    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96d2936e-4782-4bc3-b4a3-2ebb19701fdc" path="/var/lib/kubelet/pods/96d2936e-4782-4bc3-b4a3-2ebb19701fdc/volumes"
Oct 30 20:11:34.302830 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.302812    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"
Oct 30 20:11:34.397132 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397099    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-podinfo\") pod \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") "
Oct 30 20:11:34.397292 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397153    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b569370e-5fdb-4809-92f1-fcb85b98b9f7-kube-api-access-qfjxn\" (UniqueName: \"kubernetes.io/projected/b569370e-5fdb-4809-92f1-fcb85b98b9f7-kube-api-access-qfjxn\") pod \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") "
Oct 30 20:11:34.397292 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397175    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-envoy\") pod \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") "
Oct 30 20:11:34.397292 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397203    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-maas-api-xks-ca\") pod \"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f\" (UID: \"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f\") "
Oct 30 20:11:34.397292 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397234    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-workload-socket\") pod \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") "
Oct 30 20:11:34.397292 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397254    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-credential-socket\") pod \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") "
Oct 30 20:11:34.397581 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397379    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-maas-api-openshift-ca\") pod \"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f\" (UID: \"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f\") "
Oct 30 20:11:34.397581 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397417    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-token\" (UniqueName: \"kubernetes.io/projected/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-token\") pod \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") "
Oct 30 20:11:34.397581 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397463    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-workload-certs\") pod \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") "
Oct 30 20:11:34.397581 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397496    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-podinfo\") pod \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") "
Oct 30 20:11:34.397581 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397542    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-workload-socket" pod "98c1ea21-d79b-457c-95f7-45fd2b2befd7" (UID: "98c1ea21-d79b-457c-95f7-45fd2b2befd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:34.397581 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397553    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istiod-ca-cert\") pod \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") "
Oct 30 20:11:34.397983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397620    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c1ea21-d79b-457c-95f7-45fd2b2befd7-kube-api-access-zb278\" (UniqueName: \"kubernetes.io/projected/98c1ea21-d79b-457c-95f7-45fd2b2befd7-kube-api-access-zb278\") pod \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") "
Oct 30 20:11:34.397983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397682    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-envoy\") pod \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") "
Oct 30 20:11:34.397983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397754    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-workload-socket\") pod \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") "
Oct 30 20:11:34.397983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397773    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-workload-certs" pod "98c1ea21-d79b-457c-95f7-45fd2b2befd7" (UID: "98c1ea21-d79b-457c-95f7-45fd2b2befd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:34.397983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397782    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-workload-certs\") pod \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") "
Oct 30 20:11:34.397983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397829    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-data\") pod \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\" (UID: \"98c1ea21-d79b-457c-95f7-45fd2b2befd7\") "
Oct 30 20:11:34.397983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397896    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-token\" (UniqueName: \"kubernetes.io/projected/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-token\") pod \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") "
Oct 30 20:11:34.397983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397955    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istiod-ca-cert\") pod \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") "
Oct 30 20:11:34.398422 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.398017    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-kube-api-access-45gkh\" (UniqueName: \"kubernetes.io/projected/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-kube-api-access-45gkh\") pod \"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f\" (UID: \"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f\") "
Oct 30 20:11:34.398422 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.398080    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-credential-socket" pod "98c1ea21-d79b-457c-95f7-45fd2b2befd7" (UID: "98c1ea21-d79b-457c-95f7-45fd2b2befd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:34.398422 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.398105    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-credential-socket\") pod \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") "
Oct 30 20:11:34.398422 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.397924    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-maas-api-openshift-ca" pod "2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f" (UID: "2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:34.398422 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.398208    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-data\") pod \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\" (UID: \"b569370e-5fdb-4809-92f1-fcb85b98b9f7\") "
Oct 30 20:11:34.398694 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.398620    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-workload-socket" pod "b569370e-5fdb-4809-92f1-fcb85b98b9f7" (UID: "b569370e-5fdb-4809-92f1-fcb85b98b9f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:34.399579 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.399549    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.399747 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.399584    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.399747 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.399600    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.399747 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.399613    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.399747 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.399625    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.400267 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.400130    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-podinfo" pod "b569370e-5fdb-4809-92f1-fcb85b98b9f7" (UID: "b569370e-5fdb-4809-92f1-fcb85b98b9f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:11:34.400267 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.400154    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istiod-ca-cert" pod "b569370e-5fdb-4809-92f1-fcb85b98b9f7" (UID: "b569370e-5fdb-4809-92f1-fcb85b98b9f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:34.400267 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.400212    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-maas-api-xks-ca" pod "2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f" (UID: "2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:34.400267 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.400238    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b569370e-5fdb-4809-92f1-fcb85b98b9f7-kube-api-access-qfjxn" pod "b569370e-5fdb-4809-92f1-fcb85b98b9f7" (UID: "b569370e-5fdb-4809-92f1-fcb85b98b9f7"). InnerVolumeSpecName "kube-api-access-qfjxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:34.400634 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.400361    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-token" pod "98c1ea21-d79b-457c-95f7-45fd2b2befd7" (UID: "98c1ea21-d79b-457c-95f7-45fd2b2befd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:34.400634 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.400528    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-credential-socket" pod "b569370e-5fdb-4809-92f1-fcb85b98b9f7" (UID: "b569370e-5fdb-4809-92f1-fcb85b98b9f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:34.400634 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.400618    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-envoy" pod "98c1ea21-d79b-457c-95f7-45fd2b2befd7" (UID: "98c1ea21-d79b-457c-95f7-45fd2b2befd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:34.400813 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.400761    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istiod-ca-cert" pod "98c1ea21-d79b-457c-95f7-45fd2b2befd7" (UID: "98c1ea21-d79b-457c-95f7-45fd2b2befd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:34.400813 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.400767    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-data" pod "b569370e-5fdb-4809-92f1-fcb85b98b9f7" (UID: "b569370e-5fdb-4809-92f1-fcb85b98b9f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:34.401021 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.400985    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-workload-certs" pod "b569370e-5fdb-4809-92f1-fcb85b98b9f7" (UID: "b569370e-5fdb-4809-92f1-fcb85b98b9f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:34.401021 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.401002    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-podinfo" pod "98c1ea21-d79b-457c-95f7-45fd2b2befd7" (UID: "98c1ea21-d79b-457c-95f7-45fd2b2befd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:11:34.401286 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.401261    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-data" pod "98c1ea21-d79b-457c-95f7-45fd2b2befd7" (UID: "98c1ea21-d79b-457c-95f7-45fd2b2befd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:34.401352 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.401334    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-token" pod "b569370e-5fdb-4809-92f1-fcb85b98b9f7" (UID: "b569370e-5fdb-4809-92f1-fcb85b98b9f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:34.402323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.402295    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-kube-api-access-45gkh" pod "2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f" (UID: "2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f"). InnerVolumeSpecName "kube-api-access-45gkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:34.402506 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.402486    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-envoy" pod "b569370e-5fdb-4809-92f1-fcb85b98b9f7" (UID: "b569370e-5fdb-4809-92f1-fcb85b98b9f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:34.402831 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.402809    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c1ea21-d79b-457c-95f7-45fd2b2befd7-kube-api-access-zb278" pod "98c1ea21-d79b-457c-95f7-45fd2b2befd7" (UID: "98c1ea21-d79b-457c-95f7-45fd2b2befd7"). InnerVolumeSpecName "kube-api-access-zb278". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:34.501762 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.501728    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.501762 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.501760    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.501762 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.501770    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.502070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.501781    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.502070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.501792    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45gkh\" (UniqueName: \"kubernetes.io/projected/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-kube-api-access-45gkh\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.502070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.501803    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.502070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.501812    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.502070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.501820    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.502070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.501829    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfjxn\" (UniqueName: \"kubernetes.io/projected/b569370e-5fdb-4809-92f1-fcb85b98b9f7-kube-api-access-qfjxn\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.502070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.501838    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.502070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.501847    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.502070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.501855    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.502070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.501879    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.502070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.501887    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c1ea21-d79b-457c-95f7-45fd2b2befd7-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.502070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.501895    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb278\" (UniqueName: \"kubernetes.io/projected/98c1ea21-d79b-457c-95f7-45fd2b2befd7-kube-api-access-zb278\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:34.502070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:34.501904    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b569370e-5fdb-4809-92f1-fcb85b98b9f7-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:35.176506 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.176013    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8" event={"ID":"98c1ea21-d79b-457c-95f7-45fd2b2befd7","Type":"ContainerDied","Data":"04ca6a66b480524881128f4e713996a043782603396b6c7a652a7202b708e28a"}
Oct 30 20:11:35.176506 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.176112    4823 scope.go:122] "RemoveContainer" containerID="53e9419693787ca9cad508cf5d087c5af61bba27af43a9da65e78f6223821d09"
Oct 30 20:11:35.176506 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.176272    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"
Oct 30 20:11:35.180777 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.180746    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"]
Oct 30 20:11:35.181089 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.181035    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg" podUID="38b2ae10-5954-4364-83f3-7e79ee7ad4f9" containerName="maas-api" containerID="cri-o://5d995d5ef1376be27eed251096b54dfe00b662d5d7f7c2695b56c221dd7bbba6" gracePeriod=30
Oct 30 20:11:35.182638 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.182613    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"
Oct 30 20:11:35.183096 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.183071    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5" event={"ID":"2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f","Type":"ContainerDied","Data":"f5038880280fa15252f8cd6deb7eb0dd05158cf16903e2e0a7d436a13ae29ada"}
Oct 30 20:11:35.185587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.184807    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5038880280fa15252f8cd6deb7eb0dd05158cf16903e2e0a7d436a13ae29ada"
Oct 30 20:11:35.188217 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.188191    4823 generic.go:356] "Generic (PLEG): container finished" podID="b569370e-5fdb-4809-92f1-fcb85b98b9f7" containerID="5fbbfc5ebd47fc01331fb42d8e2c58790dd0b64bc844282cbacaf0da97be4376" exitCode=0
Oct 30 20:11:35.188418 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.188398    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l" event={"ID":"b569370e-5fdb-4809-92f1-fcb85b98b9f7","Type":"ContainerDied","Data":"5fbbfc5ebd47fc01331fb42d8e2c58790dd0b64bc844282cbacaf0da97be4376"}
Oct 30 20:11:35.188550 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.188533    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l" event={"ID":"b569370e-5fdb-4809-92f1-fcb85b98b9f7","Type":"ContainerDied","Data":"a5fa0f72a70b8f2fa585a0240cbd64ff2da689258165d210acd13dda87f73952"}
Oct 30 20:11:35.188706 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.188692    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"
Oct 30 20:11:35.226445 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.226422    4823 scope.go:122] "RemoveContainer" containerID="5fbbfc5ebd47fc01331fb42d8e2c58790dd0b64bc844282cbacaf0da97be4376"
Oct 30 20:11:35.273958 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.273924    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"]
Oct 30 20:11:35.280075 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.280038    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-44kv8"]
Oct 30 20:11:35.367960 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.367918    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"]
Oct 30 20:11:35.393173 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.385964    4823 scope.go:122] "RemoveContainer" containerID="5fbbfc5ebd47fc01331fb42d8e2c58790dd0b64bc844282cbacaf0da97be4376"
Oct 30 20:11:35.393173 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:11:35.388423    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fbbfc5ebd47fc01331fb42d8e2c58790dd0b64bc844282cbacaf0da97be4376\": container with ID starting with 5fbbfc5ebd47fc01331fb42d8e2c58790dd0b64bc844282cbacaf0da97be4376 not found: ID does not exist" containerID="5fbbfc5ebd47fc01331fb42d8e2c58790dd0b64bc844282cbacaf0da97be4376"
Oct 30 20:11:35.393173 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.388476    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fbbfc5ebd47fc01331fb42d8e2c58790dd0b64bc844282cbacaf0da97be4376"} err="failed to get container status \"5fbbfc5ebd47fc01331fb42d8e2c58790dd0b64bc844282cbacaf0da97be4376\": rpc error: code = NotFound desc = could not find container \"5fbbfc5ebd47fc01331fb42d8e2c58790dd0b64bc844282cbacaf0da97be4376\": container with ID starting with 5fbbfc5ebd47fc01331fb42d8e2c58790dd0b64bc844282cbacaf0da97be4376 not found: ID does not exist"
Oct 30 20:11:35.408107 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.408067    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-pbg6l"]
Oct 30 20:11:35.872481 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.872449    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:11:35.957872 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.957827    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-openshift-service-ca\") pod \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") "
Oct 30 20:11:35.958073 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.957913    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-maas-api-tls\") pod \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") "
Oct 30 20:11:35.958073 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.958027    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-metrics-certs\") pod \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") "
Oct 30 20:11:35.958217 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.958083    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-kube-api-access-j6p9d\" (UniqueName: \"kubernetes.io/projected/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-kube-api-access-j6p9d\") pod \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\" (UID: \"38b2ae10-5954-4364-83f3-7e79ee7ad4f9\") "
Oct 30 20:11:35.958333 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.958309    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-openshift-service-ca" pod "38b2ae10-5954-4364-83f3-7e79ee7ad4f9" (UID: "38b2ae10-5954-4364-83f3-7e79ee7ad4f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:35.958829 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.958807    4823 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-openshift-service-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:35.960252 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.960227    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-maas-api-tls" pod "38b2ae10-5954-4364-83f3-7e79ee7ad4f9" (UID: "38b2ae10-5954-4364-83f3-7e79ee7ad4f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:35.960813 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.960765    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-metrics-certs" pod "38b2ae10-5954-4364-83f3-7e79ee7ad4f9" (UID: "38b2ae10-5954-4364-83f3-7e79ee7ad4f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:35.961024 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.960997    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-kube-api-access-j6p9d" pod "38b2ae10-5954-4364-83f3-7e79ee7ad4f9" (UID: "38b2ae10-5954-4364-83f3-7e79ee7ad4f9"). InnerVolumeSpecName "kube-api-access-j6p9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:35.985455 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.985423    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"]
Oct 30 20:11:35.990183 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:35.990156    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-55d2e0-7mkcp"]
Oct 30 20:11:36.060773 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:36.060735    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-maas-api-tls\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:36.060773 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:36.060767    4823 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-metrics-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:36.061028 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:36.060783    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6p9d\" (UniqueName: \"kubernetes.io/projected/38b2ae10-5954-4364-83f3-7e79ee7ad4f9-kube-api-access-j6p9d\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:36.199558 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:36.199522    4823 generic.go:356] "Generic (PLEG): container finished" podID="38b2ae10-5954-4364-83f3-7e79ee7ad4f9" containerID="5d995d5ef1376be27eed251096b54dfe00b662d5d7f7c2695b56c221dd7bbba6" exitCode=0
Oct 30 20:11:36.200064 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:36.199573    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"
Oct 30 20:11:36.200064 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:36.199612    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg" event={"ID":"38b2ae10-5954-4364-83f3-7e79ee7ad4f9","Type":"ContainerDied","Data":"5d995d5ef1376be27eed251096b54dfe00b662d5d7f7c2695b56c221dd7bbba6"}
Oct 30 20:11:36.200064 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:36.199708    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg" event={"ID":"38b2ae10-5954-4364-83f3-7e79ee7ad4f9","Type":"ContainerDied","Data":"71fb9f2871ee5b9d649dd01ac41a2e5482c7d4d5dcfaf0a840d5b88c8e661910"}
Oct 30 20:11:36.200064 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:36.199745    4823 scope.go:122] "RemoveContainer" containerID="5d995d5ef1376be27eed251096b54dfe00b662d5d7f7c2695b56c221dd7bbba6"
Oct 30 20:11:36.213985 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:36.213963    4823 scope.go:122] "RemoveContainer" containerID="5d995d5ef1376be27eed251096b54dfe00b662d5d7f7c2695b56c221dd7bbba6"
Oct 30 20:11:36.214392 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:11:36.214365    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d995d5ef1376be27eed251096b54dfe00b662d5d7f7c2695b56c221dd7bbba6\": container with ID starting with 5d995d5ef1376be27eed251096b54dfe00b662d5d7f7c2695b56c221dd7bbba6 not found: ID does not exist" containerID="5d995d5ef1376be27eed251096b54dfe00b662d5d7f7c2695b56c221dd7bbba6"
Oct 30 20:11:36.214473 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:36.214402    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d995d5ef1376be27eed251096b54dfe00b662d5d7f7c2695b56c221dd7bbba6"} err="failed to get container status \"5d995d5ef1376be27eed251096b54dfe00b662d5d7f7c2695b56c221dd7bbba6\": rpc error: code = NotFound desc = could not find container \"5d995d5ef1376be27eed251096b54dfe00b662d5d7f7c2695b56c221dd7bbba6\": container with ID starting with 5d995d5ef1376be27eed251096b54dfe00b662d5d7f7c2695b56c221dd7bbba6 not found: ID does not exist"
Oct 30 20:11:36.226092 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:36.226067    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"]
Oct 30 20:11:36.232586 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:36.232561    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55d2e0-96b7d544b-wwnkg"]
Oct 30 20:11:36.280335 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:36.280300    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38b2ae10-5954-4364-83f3-7e79ee7ad4f9" path="/var/lib/kubelet/pods/38b2ae10-5954-4364-83f3-7e79ee7ad4f9/volumes"
Oct 30 20:11:36.280902 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:36.280842    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a5c978-c96f-44c0-ac3a-828d454fae89" path="/var/lib/kubelet/pods/45a5c978-c96f-44c0-ac3a-828d454fae89/volumes"
Oct 30 20:11:36.281343 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:36.281324    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" path="/var/lib/kubelet/pods/98c1ea21-d79b-457c-95f7-45fd2b2befd7/volumes"
Oct 30 20:11:36.282224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:36.282207    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b569370e-5fdb-4809-92f1-fcb85b98b9f7" path="/var/lib/kubelet/pods/b569370e-5fdb-4809-92f1-fcb85b98b9f7/volumes"
Oct 30 20:11:37.447081 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:37.447042    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"]
Oct 30 20:11:37.452092 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:37.452062    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a7bb6c-rb75k"]
Oct 30 20:11:38.281132 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:38.281101    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39484f68-3e11-417e-b0a7-d562ad75a758" path="/var/lib/kubelet/pods/39484f68-3e11-417e-b0a7-d562ad75a758/volumes"
Oct 30 20:11:39.931265 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:39.931221    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:39.944585 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:39.944552    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:40.377341 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.377298    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:11:40.478886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.477487    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"]
Oct 30 20:11:40.479275 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479245    4823 cpu_manager.go:414] "containerMap: removing container" podUID="38b2ae10-5954-4364-83f3-7e79ee7ad4f9" containerName="maas-api"
Oct 30 20:11:40.479275 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479275    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b2ae10-5954-4364-83f3-7e79ee7ad4f9" containerName="maas-api"
Oct 30 20:11:40.479576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479292    4823 cpu_manager.go:414] "containerMap: removing container" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy"
Oct 30 20:11:40.479576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479300    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy"
Oct 30 20:11:40.479576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479330    4823 cpu_manager.go:414] "containerMap: removing container" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy"
Oct 30 20:11:40.479576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479339    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy"
Oct 30 20:11:40.479576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479349    4823 cpu_manager.go:414] "containerMap: removing container" podUID="3d91f3d0-a115-4e98-a683-a8fd017d89d9" containerName="revoke-keys"
Oct 30 20:11:40.479576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479357    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d91f3d0-a115-4e98-a683-a8fd017d89d9" containerName="revoke-keys"
Oct 30 20:11:40.479576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479374    4823 cpu_manager.go:414] "containerMap: removing container" podUID="96d2936e-4782-4bc3-b4a3-2ebb19701fdc" containerName="maas-api"
Oct 30 20:11:40.479576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479381    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d2936e-4782-4bc3-b4a3-2ebb19701fdc" containerName="maas-api"
Oct 30 20:11:40.479576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479418    4823 cpu_manager.go:414] "containerMap: removing container" podUID="2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f" containerName="revoke-keys"
Oct 30 20:11:40.479576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479425    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f" containerName="revoke-keys"
Oct 30 20:11:40.479576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479473    4823 cpu_manager.go:414] "containerMap: removing container" podUID="b569370e-5fdb-4809-92f1-fcb85b98b9f7" containerName="istio-proxy"
Oct 30 20:11:40.479576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479481    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b569370e-5fdb-4809-92f1-fcb85b98b9f7" containerName="istio-proxy"
Oct 30 20:11:40.480339 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479651    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d2936e-4782-4bc3-b4a3-2ebb19701fdc" containerName="maas-api"
Oct 30 20:11:40.480339 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479668    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f" containerName="revoke-keys"
Oct 30 20:11:40.480339 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479687    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="b569370e-5fdb-4809-92f1-fcb85b98b9f7" containerName="istio-proxy"
Oct 30 20:11:40.480339 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479698    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b2ae10-5954-4364-83f3-7e79ee7ad4f9" containerName="maas-api"
Oct 30 20:11:40.480339 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479707    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy"
Oct 30 20:11:40.480339 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479720    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c1ea21-d79b-457c-95f7-45fd2b2befd7" containerName="istio-proxy"
Oct 30 20:11:40.480339 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.479732    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d91f3d0-a115-4e98-a683-a8fd017d89d9" containerName="revoke-keys"
Oct 30 20:11:40.507181 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.503036    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"]
Oct 30 20:11:40.507181 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.503088    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"]
Oct 30 20:11:40.507181 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.504243    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"
Oct 30 20:11:40.508316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.508280    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:11:40.508578 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.508459    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:11:40.519037 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.515236    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"]
Oct 30 20:11:40.519037 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.515448    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"
Oct 30 20:11:40.582742 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.582699    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4feaeb6-8155-40f0-b659-acc2d1953d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbfb04aa-qd7xt\" (UID: \"f4feaeb6-8155-40f0-b659-acc2d1953d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"
Oct 30 20:11:40.582947 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.582759    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glxfv\" (UniqueName: \"kubernetes.io/projected/ad764bc6-95ee-432f-940b-c56c4b2cea63-kube-api-access-glxfv\") pod \"maas-api-revoke-sub-3623be56-85h9b\" (UID: \"ad764bc6-95ee-432f-940b-c56c4b2cea63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"
Oct 30 20:11:40.583043 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.582965    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad764bc6-95ee-432f-940b-c56c4b2cea63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3623be56-85h9b\" (UID: \"ad764bc6-95ee-432f-940b-c56c4b2cea63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"
Oct 30 20:11:40.583083 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.583061    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4feaeb6-8155-40f0-b659-acc2d1953d4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbfb04aa-qd7xt\" (UID: \"f4feaeb6-8155-40f0-b659-acc2d1953d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"
Oct 30 20:11:40.583118 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.583092    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rtl2\" (UniqueName: \"kubernetes.io/projected/f4feaeb6-8155-40f0-b659-acc2d1953d4f-kube-api-access-2rtl2\") pod \"maas-api-revoke-sub-bbfb04aa-qd7xt\" (UID: \"f4feaeb6-8155-40f0-b659-acc2d1953d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"
Oct 30 20:11:40.583294 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.583276    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad764bc6-95ee-432f-940b-c56c4b2cea63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3623be56-85h9b\" (UID: \"ad764bc6-95ee-432f-940b-c56c4b2cea63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"
Oct 30 20:11:40.684295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.684199    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad764bc6-95ee-432f-940b-c56c4b2cea63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3623be56-85h9b\" (UID: \"ad764bc6-95ee-432f-940b-c56c4b2cea63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"
Oct 30 20:11:40.684295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.684264    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4feaeb6-8155-40f0-b659-acc2d1953d4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbfb04aa-qd7xt\" (UID: \"f4feaeb6-8155-40f0-b659-acc2d1953d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"
Oct 30 20:11:40.684295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.684291    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rtl2\" (UniqueName: \"kubernetes.io/projected/f4feaeb6-8155-40f0-b659-acc2d1953d4f-kube-api-access-2rtl2\") pod \"maas-api-revoke-sub-bbfb04aa-qd7xt\" (UID: \"f4feaeb6-8155-40f0-b659-acc2d1953d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"
Oct 30 20:11:40.684592 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.684493    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad764bc6-95ee-432f-940b-c56c4b2cea63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3623be56-85h9b\" (UID: \"ad764bc6-95ee-432f-940b-c56c4b2cea63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"
Oct 30 20:11:40.684692 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.684663    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4feaeb6-8155-40f0-b659-acc2d1953d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbfb04aa-qd7xt\" (UID: \"f4feaeb6-8155-40f0-b659-acc2d1953d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"
Oct 30 20:11:40.684761 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.684714    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glxfv\" (UniqueName: \"kubernetes.io/projected/ad764bc6-95ee-432f-940b-c56c4b2cea63-kube-api-access-glxfv\") pod \"maas-api-revoke-sub-3623be56-85h9b\" (UID: \"ad764bc6-95ee-432f-940b-c56c4b2cea63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"
Oct 30 20:11:40.685008 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.684981    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad764bc6-95ee-432f-940b-c56c4b2cea63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3623be56-85h9b\" (UID: \"ad764bc6-95ee-432f-940b-c56c4b2cea63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"
Oct 30 20:11:40.685524 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.685498    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4feaeb6-8155-40f0-b659-acc2d1953d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbfb04aa-qd7xt\" (UID: \"f4feaeb6-8155-40f0-b659-acc2d1953d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"
Oct 30 20:11:40.686966 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.686941    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad764bc6-95ee-432f-940b-c56c4b2cea63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3623be56-85h9b\" (UID: \"ad764bc6-95ee-432f-940b-c56c4b2cea63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"
Oct 30 20:11:40.687371 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.687348    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4feaeb6-8155-40f0-b659-acc2d1953d4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbfb04aa-qd7xt\" (UID: \"f4feaeb6-8155-40f0-b659-acc2d1953d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"
Oct 30 20:11:40.698639 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.698607    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glxfv\" (UniqueName: \"kubernetes.io/projected/ad764bc6-95ee-432f-940b-c56c4b2cea63-kube-api-access-glxfv\") pod \"maas-api-revoke-sub-3623be56-85h9b\" (UID: \"ad764bc6-95ee-432f-940b-c56c4b2cea63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"
Oct 30 20:11:40.703132 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.703106    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rtl2\" (UniqueName: \"kubernetes.io/projected/f4feaeb6-8155-40f0-b659-acc2d1953d4f-kube-api-access-2rtl2\") pod \"maas-api-revoke-sub-bbfb04aa-qd7xt\" (UID: \"f4feaeb6-8155-40f0-b659-acc2d1953d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"
Oct 30 20:11:40.841933 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.841895    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"
Oct 30 20:11:40.853737 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:40.853709    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"
Oct 30 20:11:41.226985 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:11:41.226952    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad764bc6_95ee_432f_940b_c56c4b2cea63.slice/crio-2afdfee4470fba0604513b43b50761623087341b87e6ab3ab164d3f8e681aee3 WatchSource:0}: Error finding container 2afdfee4470fba0604513b43b50761623087341b87e6ab3ab164d3f8e681aee3: Status 404 returned error can't find the container with id 2afdfee4470fba0604513b43b50761623087341b87e6ab3ab164d3f8e681aee3
Oct 30 20:11:41.227498 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:41.227140    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"]
Oct 30 20:11:41.257683 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:41.257662    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"]
Oct 30 20:11:41.265389 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:11:41.265362    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4feaeb6_8155_40f0_b659_acc2d1953d4f.slice/crio-9ffb1c749df849f11704887b9fd0b303acf9b651f86ba655a62cc4a3f7f84d56 WatchSource:0}: Error finding container 9ffb1c749df849f11704887b9fd0b303acf9b651f86ba655a62cc4a3f7f84d56: Status 404 returned error can't find the container with id 9ffb1c749df849f11704887b9fd0b303acf9b651f86ba655a62cc4a3f7f84d56
Oct 30 20:11:41.442884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:41.440896    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b" event={"ID":"ad764bc6-95ee-432f-940b-c56c4b2cea63","Type":"ContainerStarted","Data":"2afdfee4470fba0604513b43b50761623087341b87e6ab3ab164d3f8e681aee3"}
Oct 30 20:11:41.443107 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:41.443073    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt" event={"ID":"f4feaeb6-8155-40f0-b659-acc2d1953d4f","Type":"ContainerStarted","Data":"9ffb1c749df849f11704887b9fd0b303acf9b651f86ba655a62cc4a3f7f84d56"}
Oct 30 20:11:42.576939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:42.573518    4823 generic.go:356] "Generic (PLEG): container finished" podID="ad764bc6-95ee-432f-940b-c56c4b2cea63" containerID="a6c419e795fc117e9d507b750e7b8385b00991a96439ab3b30218b2609903e96" exitCode=0
Oct 30 20:11:42.576939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:42.573660    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b" event={"ID":"ad764bc6-95ee-432f-940b-c56c4b2cea63","Type":"ContainerDied","Data":"a6c419e795fc117e9d507b750e7b8385b00991a96439ab3b30218b2609903e96"}
Oct 30 20:11:42.616885 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:42.616715    4823 generic.go:356] "Generic (PLEG): container finished" podID="f4feaeb6-8155-40f0-b659-acc2d1953d4f" containerID="c55ac7a0808475aca4de47c34ea7abde619e2a46b8b9f91f5d7846e75a89db72" exitCode=0
Oct 30 20:11:42.616885 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:42.616814    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt" event={"ID":"f4feaeb6-8155-40f0-b659-acc2d1953d4f","Type":"ContainerDied","Data":"c55ac7a0808475aca4de47c34ea7abde619e2a46b8b9f91f5d7846e75a89db72"}
Oct 30 20:11:42.881422 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:42.881383    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"]
Oct 30 20:11:42.882376 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:42.882342    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g" podUID="236fd49d-b067-40ba-83ab-def38b3b0862" containerName="maas-api" containerID="cri-o://284f88e9291f588bfdf20626fab8ca83026f0c6b93adeefd874a05ace3ebcdc2" gracePeriod=30
Oct 30 20:11:43.415323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.415296    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:11:43.571576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.571539    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/236fd49d-b067-40ba-83ab-def38b3b0862-kube-api-access-ftzxg\" (UniqueName: \"kubernetes.io/projected/236fd49d-b067-40ba-83ab-def38b3b0862-kube-api-access-ftzxg\") pod \"236fd49d-b067-40ba-83ab-def38b3b0862\" (UID: \"236fd49d-b067-40ba-83ab-def38b3b0862\") "
Oct 30 20:11:43.571766 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.571750    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/236fd49d-b067-40ba-83ab-def38b3b0862-metrics-certs\" (UniqueName: \"kubernetes.io/secret/236fd49d-b067-40ba-83ab-def38b3b0862-metrics-certs\") pod \"236fd49d-b067-40ba-83ab-def38b3b0862\" (UID: \"236fd49d-b067-40ba-83ab-def38b3b0862\") "
Oct 30 20:11:43.571857 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.571787    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/236fd49d-b067-40ba-83ab-def38b3b0862-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/236fd49d-b067-40ba-83ab-def38b3b0862-openshift-service-ca\") pod \"236fd49d-b067-40ba-83ab-def38b3b0862\" (UID: \"236fd49d-b067-40ba-83ab-def38b3b0862\") "
Oct 30 20:11:43.572015 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.571998    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/236fd49d-b067-40ba-83ab-def38b3b0862-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/236fd49d-b067-40ba-83ab-def38b3b0862-maas-api-tls\") pod \"236fd49d-b067-40ba-83ab-def38b3b0862\" (UID: \"236fd49d-b067-40ba-83ab-def38b3b0862\") "
Oct 30 20:11:43.572257 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.572221    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/236fd49d-b067-40ba-83ab-def38b3b0862-openshift-service-ca" pod "236fd49d-b067-40ba-83ab-def38b3b0862" (UID: "236fd49d-b067-40ba-83ab-def38b3b0862"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:43.572656 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.572635    4823 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/236fd49d-b067-40ba-83ab-def38b3b0862-openshift-service-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:43.574257 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.574230    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/236fd49d-b067-40ba-83ab-def38b3b0862-maas-api-tls" pod "236fd49d-b067-40ba-83ab-def38b3b0862" (UID: "236fd49d-b067-40ba-83ab-def38b3b0862"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:43.574494 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.574470    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/236fd49d-b067-40ba-83ab-def38b3b0862-kube-api-access-ftzxg" pod "236fd49d-b067-40ba-83ab-def38b3b0862" (UID: "236fd49d-b067-40ba-83ab-def38b3b0862"). InnerVolumeSpecName "kube-api-access-ftzxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:43.574581 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.574498    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/236fd49d-b067-40ba-83ab-def38b3b0862-metrics-certs" pod "236fd49d-b067-40ba-83ab-def38b3b0862" (UID: "236fd49d-b067-40ba-83ab-def38b3b0862"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:43.626544 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.626504    4823 generic.go:356] "Generic (PLEG): container finished" podID="236fd49d-b067-40ba-83ab-def38b3b0862" containerID="284f88e9291f588bfdf20626fab8ca83026f0c6b93adeefd874a05ace3ebcdc2" exitCode=0
Oct 30 20:11:43.627036 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.626555    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"
Oct 30 20:11:43.627036 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.626596    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g" event={"ID":"236fd49d-b067-40ba-83ab-def38b3b0862","Type":"ContainerDied","Data":"284f88e9291f588bfdf20626fab8ca83026f0c6b93adeefd874a05ace3ebcdc2"}
Oct 30 20:11:43.627036 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.626653    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g" event={"ID":"236fd49d-b067-40ba-83ab-def38b3b0862","Type":"ContainerDied","Data":"90449f36c9613877ad139831d2c00abe95de0e0aac365d8e3b5e1e5ab6c46bec"}
Oct 30 20:11:43.627036 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.626739    4823 scope.go:122] "RemoveContainer" containerID="284f88e9291f588bfdf20626fab8ca83026f0c6b93adeefd874a05ace3ebcdc2"
Oct 30 20:11:43.649151 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.649116    4823 scope.go:122] "RemoveContainer" containerID="284f88e9291f588bfdf20626fab8ca83026f0c6b93adeefd874a05ace3ebcdc2"
Oct 30 20:11:43.649616 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:11:43.649584    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"284f88e9291f588bfdf20626fab8ca83026f0c6b93adeefd874a05ace3ebcdc2\": container with ID starting with 284f88e9291f588bfdf20626fab8ca83026f0c6b93adeefd874a05ace3ebcdc2 not found: ID does not exist" containerID="284f88e9291f588bfdf20626fab8ca83026f0c6b93adeefd874a05ace3ebcdc2"
Oct 30 20:11:43.649721 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.649626    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"284f88e9291f588bfdf20626fab8ca83026f0c6b93adeefd874a05ace3ebcdc2"} err="failed to get container status \"284f88e9291f588bfdf20626fab8ca83026f0c6b93adeefd874a05ace3ebcdc2\": rpc error: code = NotFound desc = could not find container \"284f88e9291f588bfdf20626fab8ca83026f0c6b93adeefd874a05ace3ebcdc2\": container with ID starting with 284f88e9291f588bfdf20626fab8ca83026f0c6b93adeefd874a05ace3ebcdc2 not found: ID does not exist"
Oct 30 20:11:43.671884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.671829    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"]
Oct 30 20:11:43.675047 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.675011    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftzxg\" (UniqueName: \"kubernetes.io/projected/236fd49d-b067-40ba-83ab-def38b3b0862-kube-api-access-ftzxg\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:43.675047 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.675045    4823 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/236fd49d-b067-40ba-83ab-def38b3b0862-metrics-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:43.675275 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.675059    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/236fd49d-b067-40ba-83ab-def38b3b0862-maas-api-tls\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:43.727673 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.727621    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a4f02-5578cf9988-s4f2g"]
Oct 30 20:11:43.839911 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.836710    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"
Oct 30 20:11:43.905495 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.905024    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4feaeb6-8155-40f0-b659-acc2d1953d4f-kube-api-access-2rtl2\" (UniqueName: \"kubernetes.io/projected/f4feaeb6-8155-40f0-b659-acc2d1953d4f-kube-api-access-2rtl2\") pod \"f4feaeb6-8155-40f0-b659-acc2d1953d4f\" (UID: \"f4feaeb6-8155-40f0-b659-acc2d1953d4f\") "
Oct 30 20:11:43.905495 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.905091    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4feaeb6-8155-40f0-b659-acc2d1953d4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4feaeb6-8155-40f0-b659-acc2d1953d4f-maas-api-xks-ca\") pod \"f4feaeb6-8155-40f0-b659-acc2d1953d4f\" (UID: \"f4feaeb6-8155-40f0-b659-acc2d1953d4f\") "
Oct 30 20:11:43.906911 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.905756    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4feaeb6-8155-40f0-b659-acc2d1953d4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4feaeb6-8155-40f0-b659-acc2d1953d4f-maas-api-openshift-ca\") pod \"f4feaeb6-8155-40f0-b659-acc2d1953d4f\" (UID: \"f4feaeb6-8155-40f0-b659-acc2d1953d4f\") "
Oct 30 20:11:43.907251 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.907229    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4feaeb6-8155-40f0-b659-acc2d1953d4f-maas-api-openshift-ca" pod "f4feaeb6-8155-40f0-b659-acc2d1953d4f" (UID: "f4feaeb6-8155-40f0-b659-acc2d1953d4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:43.917278 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.917253    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"]
Oct 30 20:11:43.918514 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.918489    4823 cpu_manager.go:414] "containerMap: removing container" podUID="236fd49d-b067-40ba-83ab-def38b3b0862" containerName="maas-api"
Oct 30 20:11:43.918675 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.918661    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="236fd49d-b067-40ba-83ab-def38b3b0862" containerName="maas-api"
Oct 30 20:11:43.918853 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.918826    4823 cpu_manager.go:414] "containerMap: removing container" podUID="f4feaeb6-8155-40f0-b659-acc2d1953d4f" containerName="revoke-keys"
Oct 30 20:11:43.918977 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.918962    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4feaeb6-8155-40f0-b659-acc2d1953d4f" containerName="revoke-keys"
Oct 30 20:11:43.919346 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.919314    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="236fd49d-b067-40ba-83ab-def38b3b0862" containerName="maas-api"
Oct 30 20:11:43.919455 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.919443    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4feaeb6-8155-40f0-b659-acc2d1953d4f" containerName="revoke-keys"
Oct 30 20:11:43.924778 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.924697    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"
Oct 30 20:11:43.927270 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.927197    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4feaeb6-8155-40f0-b659-acc2d1953d4f-maas-api-xks-ca" pod "f4feaeb6-8155-40f0-b659-acc2d1953d4f" (UID: "f4feaeb6-8155-40f0-b659-acc2d1953d4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:43.929341 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.929320    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"]
Oct 30 20:11:43.935715 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.935684    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4feaeb6-8155-40f0-b659-acc2d1953d4f-kube-api-access-2rtl2" pod "f4feaeb6-8155-40f0-b659-acc2d1953d4f" (UID: "f4feaeb6-8155-40f0-b659-acc2d1953d4f"). InnerVolumeSpecName "kube-api-access-2rtl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:43.989109 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:43.989075    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"
Oct 30 20:11:44.007790 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.007760    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad764bc6-95ee-432f-940b-c56c4b2cea63-kube-api-access-glxfv\" (UniqueName: \"kubernetes.io/projected/ad764bc6-95ee-432f-940b-c56c4b2cea63-kube-api-access-glxfv\") pod \"ad764bc6-95ee-432f-940b-c56c4b2cea63\" (UID: \"ad764bc6-95ee-432f-940b-c56c4b2cea63\") "
Oct 30 20:11:44.008028 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.008010    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad764bc6-95ee-432f-940b-c56c4b2cea63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad764bc6-95ee-432f-940b-c56c4b2cea63-maas-api-openshift-ca\") pod \"ad764bc6-95ee-432f-940b-c56c4b2cea63\" (UID: \"ad764bc6-95ee-432f-940b-c56c4b2cea63\") "
Oct 30 20:11:44.008164 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.008139    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad764bc6-95ee-432f-940b-c56c4b2cea63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad764bc6-95ee-432f-940b-c56c4b2cea63-maas-api-xks-ca\") pod \"ad764bc6-95ee-432f-940b-c56c4b2cea63\" (UID: \"ad764bc6-95ee-432f-940b-c56c4b2cea63\") "
Oct 30 20:11:44.008397 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.008363    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad764bc6-95ee-432f-940b-c56c4b2cea63-maas-api-openshift-ca" pod "ad764bc6-95ee-432f-940b-c56c4b2cea63" (UID: "ad764bc6-95ee-432f-940b-c56c4b2cea63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:44.008620 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.008595    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff4qv\" (UniqueName: \"kubernetes.io/projected/86101de1-ddcb-4d84-b487-7221a42ee4db-kube-api-access-ff4qv\") pod \"maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p\" (UID: \"86101de1-ddcb-4d84-b487-7221a42ee4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"
Oct 30 20:11:44.008811 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.008793    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86101de1-ddcb-4d84-b487-7221a42ee4db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p\" (UID: \"86101de1-ddcb-4d84-b487-7221a42ee4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"
Oct 30 20:11:44.008980 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.008832    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86101de1-ddcb-4d84-b487-7221a42ee4db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p\" (UID: \"86101de1-ddcb-4d84-b487-7221a42ee4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"
Oct 30 20:11:44.009060 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.009007    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad764bc6-95ee-432f-940b-c56c4b2cea63-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:44.009060 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.009026    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rtl2\" (UniqueName: \"kubernetes.io/projected/f4feaeb6-8155-40f0-b659-acc2d1953d4f-kube-api-access-2rtl2\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:44.009060 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.009036    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4feaeb6-8155-40f0-b659-acc2d1953d4f-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:44.009060 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.009045    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4feaeb6-8155-40f0-b659-acc2d1953d4f-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:44.010109 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.010088    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad764bc6-95ee-432f-940b-c56c4b2cea63-kube-api-access-glxfv" pod "ad764bc6-95ee-432f-940b-c56c4b2cea63" (UID: "ad764bc6-95ee-432f-940b-c56c4b2cea63"). InnerVolumeSpecName "kube-api-access-glxfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:44.010348 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.010322    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad764bc6-95ee-432f-940b-c56c4b2cea63-maas-api-xks-ca" pod "ad764bc6-95ee-432f-940b-c56c4b2cea63" (UID: "ad764bc6-95ee-432f-940b-c56c4b2cea63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:44.111883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.111833    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff4qv\" (UniqueName: \"kubernetes.io/projected/86101de1-ddcb-4d84-b487-7221a42ee4db-kube-api-access-ff4qv\") pod \"maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p\" (UID: \"86101de1-ddcb-4d84-b487-7221a42ee4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"
Oct 30 20:11:44.112097 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.112075    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86101de1-ddcb-4d84-b487-7221a42ee4db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p\" (UID: \"86101de1-ddcb-4d84-b487-7221a42ee4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"
Oct 30 20:11:44.112166 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.112134    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86101de1-ddcb-4d84-b487-7221a42ee4db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p\" (UID: \"86101de1-ddcb-4d84-b487-7221a42ee4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"
Oct 30 20:11:44.112338 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.112323    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad764bc6-95ee-432f-940b-c56c4b2cea63-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:44.112408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.112345    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glxfv\" (UniqueName: \"kubernetes.io/projected/ad764bc6-95ee-432f-940b-c56c4b2cea63-kube-api-access-glxfv\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:44.113068 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.113038    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86101de1-ddcb-4d84-b487-7221a42ee4db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p\" (UID: \"86101de1-ddcb-4d84-b487-7221a42ee4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"
Oct 30 20:11:44.114910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.114884    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86101de1-ddcb-4d84-b487-7221a42ee4db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p\" (UID: \"86101de1-ddcb-4d84-b487-7221a42ee4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"
Oct 30 20:11:44.129495 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.129465    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff4qv\" (UniqueName: \"kubernetes.io/projected/86101de1-ddcb-4d84-b487-7221a42ee4db-kube-api-access-ff4qv\") pod \"maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p\" (UID: \"86101de1-ddcb-4d84-b487-7221a42ee4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"
Oct 30 20:11:44.296939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.292121    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"
Oct 30 20:11:44.375885 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.374722    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="236fd49d-b067-40ba-83ab-def38b3b0862" path="/var/lib/kubelet/pods/236fd49d-b067-40ba-83ab-def38b3b0862/volumes"
Oct 30 20:11:44.635701 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.635672    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"
Oct 30 20:11:44.636139 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.635699    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b" event={"ID":"ad764bc6-95ee-432f-940b-c56c4b2cea63","Type":"ContainerDied","Data":"2afdfee4470fba0604513b43b50761623087341b87e6ab3ab164d3f8e681aee3"}
Oct 30 20:11:44.636139 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.635755    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2afdfee4470fba0604513b43b50761623087341b87e6ab3ab164d3f8e681aee3"
Oct 30 20:11:44.637214 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.637193    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"
Oct 30 20:11:44.637331 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.637241    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt" event={"ID":"f4feaeb6-8155-40f0-b659-acc2d1953d4f","Type":"ContainerDied","Data":"9ffb1c749df849f11704887b9fd0b303acf9b651f86ba655a62cc4a3f7f84d56"}
Oct 30 20:11:44.637331 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.637287    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ffb1c749df849f11704887b9fd0b303acf9b651f86ba655a62cc4a3f7f84d56"
Oct 30 20:11:44.742059 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:11:44.742017    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86101de1_ddcb_4d84_b487_7221a42ee4db.slice/crio-0448f81451eb5882a32d0f54b9922df7d0313e202b8a62a699124f7c542287fb WatchSource:0}: Error finding container 0448f81451eb5882a32d0f54b9922df7d0313e202b8a62a699124f7c542287fb: Status 404 returned error can't find the container with id 0448f81451eb5882a32d0f54b9922df7d0313e202b8a62a699124f7c542287fb
Oct 30 20:11:44.742242 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.742217    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"]
Oct 30 20:11:44.751082 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.751056    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"]
Oct 30 20:11:44.751956 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.751935    4823 cpu_manager.go:414] "containerMap: removing container" podUID="ad764bc6-95ee-432f-940b-c56c4b2cea63" containerName="revoke-keys"
Oct 30 20:11:44.751956 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.751955    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad764bc6-95ee-432f-940b-c56c4b2cea63" containerName="revoke-keys"
Oct 30 20:11:44.752185 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.752171    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad764bc6-95ee-432f-940b-c56c4b2cea63" containerName="revoke-keys"
Oct 30 20:11:44.758075 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.755795    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"
Oct 30 20:11:44.770963 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.768049    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"]
Oct 30 20:11:44.828143 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.828107    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f\" (UID: \"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"
Oct 30 20:11:44.828327 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.828311    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f\" (UID: \"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"
Oct 30 20:11:44.828401 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.828348    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r2fn\" (UniqueName: \"kubernetes.io/projected/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-kube-api-access-5r2fn\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f\" (UID: \"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"
Oct 30 20:11:44.929581 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.929498    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f\" (UID: \"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"
Oct 30 20:11:44.929581 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.929560    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r2fn\" (UniqueName: \"kubernetes.io/projected/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-kube-api-access-5r2fn\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f\" (UID: \"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"
Oct 30 20:11:44.929775 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.929744    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f\" (UID: \"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"
Oct 30 20:11:44.930271 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.930251    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f\" (UID: \"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"
Oct 30 20:11:44.932185 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.932157    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f\" (UID: \"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"
Oct 30 20:11:44.940840 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:44.940815    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r2fn\" (UniqueName: \"kubernetes.io/projected/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-kube-api-access-5r2fn\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f\" (UID: \"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"
Oct 30 20:11:45.077902 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:45.077852    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"
Oct 30 20:11:45.374971 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:45.374938    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"]
Oct 30 20:11:45.375416 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:11:45.375388    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f51db7_b2e9_464c_9dd4_1e0c9875fd9d.slice/crio-8d2516139751f5f5b0097bcdc94bc926feb19b845cae41e6b7d5102fdcf56ebb WatchSource:0}: Error finding container 8d2516139751f5f5b0097bcdc94bc926feb19b845cae41e6b7d5102fdcf56ebb: Status 404 returned error can't find the container with id 8d2516139751f5f5b0097bcdc94bc926feb19b845cae41e6b7d5102fdcf56ebb
Oct 30 20:11:45.648689 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:45.648562    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p" event={"ID":"86101de1-ddcb-4d84-b487-7221a42ee4db","Type":"ContainerStarted","Data":"e2abc4eaecb52061e711df5a72c884318de6567ceef0a38e09e55a5ef59d3056"}
Oct 30 20:11:45.649178 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:45.648701    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p" event={"ID":"86101de1-ddcb-4d84-b487-7221a42ee4db","Type":"ContainerStarted","Data":"0448f81451eb5882a32d0f54b9922df7d0313e202b8a62a699124f7c542287fb"}
Oct 30 20:11:45.650032 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:45.650002    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f" event={"ID":"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d","Type":"ContainerStarted","Data":"8d2516139751f5f5b0097bcdc94bc926feb19b845cae41e6b7d5102fdcf56ebb"}
Oct 30 20:11:45.669013 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:45.668964    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p" podStartSLOduration=2.668950759 podStartE2EDuration="2.668950759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:45.665525625 +0000 UTC m=+2505.558139949" watchObservedRunningTime="2026-10-30 20:11:45.668950759 +0000 UTC m=+2505.561565083"
Oct 30 20:11:46.210530 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:46.210440    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"]
Oct 30 20:11:46.210751 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:46.210701    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq" podUID="04c51f58-c2b5-4cef-858e-512b42dba670" containerName="istio-proxy" containerID="cri-o://0d9aab1ff4d48482600f093963ef51da5b0d9e6dcf8cdc39962226d28cc60033" gracePeriod=30
Oct 30 20:11:46.662951 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:46.662906    4823 generic.go:356] "Generic (PLEG): container finished" podID="86101de1-ddcb-4d84-b487-7221a42ee4db" containerID="e2abc4eaecb52061e711df5a72c884318de6567ceef0a38e09e55a5ef59d3056" exitCode=0
Oct 30 20:11:46.663390 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:46.662990    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p" event={"ID":"86101de1-ddcb-4d84-b487-7221a42ee4db","Type":"ContainerDied","Data":"e2abc4eaecb52061e711df5a72c884318de6567ceef0a38e09e55a5ef59d3056"}
Oct 30 20:11:46.664726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:46.664701    4823 generic.go:356] "Generic (PLEG): container finished" podID="c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d" containerID="2339a8896d9265976052df5a6f28d7e9ddc53d71e0ceff8acb080b3e7c8b7c1a" exitCode=0
Oct 30 20:11:46.664824 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:46.664781    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f" event={"ID":"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d","Type":"ContainerDied","Data":"2339a8896d9265976052df5a6f28d7e9ddc53d71e0ceff8acb080b3e7c8b7c1a"}
Oct 30 20:11:46.829556 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:46.829525    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"]
Oct 30 20:11:46.834210 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:46.834185    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"
Oct 30 20:11:46.846186 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:46.846127    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"]
Oct 30 20:11:46.921220 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:46.921137    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pzmj\" (UniqueName: \"kubernetes.io/projected/1aed90ad-aaaf-418d-982d-58ee684f1514-kube-api-access-4pzmj\") pod \"maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq\" (UID: \"1aed90ad-aaaf-418d-982d-58ee684f1514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"
Oct 30 20:11:46.921379 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:46.921356    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aed90ad-aaaf-418d-982d-58ee684f1514-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq\" (UID: \"1aed90ad-aaaf-418d-982d-58ee684f1514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"
Oct 30 20:11:46.921469 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:46.921453    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aed90ad-aaaf-418d-982d-58ee684f1514-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq\" (UID: \"1aed90ad-aaaf-418d-982d-58ee684f1514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"
Oct 30 20:11:47.022780 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:47.022741    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aed90ad-aaaf-418d-982d-58ee684f1514-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq\" (UID: \"1aed90ad-aaaf-418d-982d-58ee684f1514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"
Oct 30 20:11:47.022981 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:47.022834    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aed90ad-aaaf-418d-982d-58ee684f1514-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq\" (UID: \"1aed90ad-aaaf-418d-982d-58ee684f1514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"
Oct 30 20:11:47.023056 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:47.023002    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pzmj\" (UniqueName: \"kubernetes.io/projected/1aed90ad-aaaf-418d-982d-58ee684f1514-kube-api-access-4pzmj\") pod \"maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq\" (UID: \"1aed90ad-aaaf-418d-982d-58ee684f1514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"
Oct 30 20:11:47.023459 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:47.023428    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aed90ad-aaaf-418d-982d-58ee684f1514-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq\" (UID: \"1aed90ad-aaaf-418d-982d-58ee684f1514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"
Oct 30 20:11:47.025410 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:47.025390    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aed90ad-aaaf-418d-982d-58ee684f1514-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq\" (UID: \"1aed90ad-aaaf-418d-982d-58ee684f1514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"
Oct 30 20:11:47.036921 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:47.036900    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pzmj\" (UniqueName: \"kubernetes.io/projected/1aed90ad-aaaf-418d-982d-58ee684f1514-kube-api-access-4pzmj\") pod \"maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq\" (UID: \"1aed90ad-aaaf-418d-982d-58ee684f1514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"
Oct 30 20:11:47.152305 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:47.152266    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"
Oct 30 20:11:47.453928 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:47.453901    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"]
Oct 30 20:11:47.454518 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:11:47.454494    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aed90ad_aaaf_418d_982d_58ee684f1514.slice/crio-e2f757e953b232e3062f99d4d73c22c579be12c2aaf7e209d639b90e500035bc WatchSource:0}: Error finding container e2f757e953b232e3062f99d4d73c22c579be12c2aaf7e209d639b90e500035bc: Status 404 returned error can't find the container with id e2f757e953b232e3062f99d4d73c22c579be12c2aaf7e209d639b90e500035bc
Oct 30 20:11:47.677590 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:47.677551    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq" event={"ID":"1aed90ad-aaaf-418d-982d-58ee684f1514","Type":"ContainerStarted","Data":"e2f757e953b232e3062f99d4d73c22c579be12c2aaf7e209d639b90e500035bc"}
Oct 30 20:11:47.905803 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:47.905776    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"
Oct 30 20:11:47.917993 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:47.917973    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"
Oct 30 20:11:47.973323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:47.973283    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"]
Oct 30 20:11:48.040840 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.040815    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-kube-api-access-5r2fn\" (UniqueName: \"kubernetes.io/projected/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-kube-api-access-5r2fn\") pod \"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d\" (UID: \"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d\") "
Oct 30 20:11:48.040997 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.040854    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86101de1-ddcb-4d84-b487-7221a42ee4db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86101de1-ddcb-4d84-b487-7221a42ee4db-maas-api-openshift-ca\") pod \"86101de1-ddcb-4d84-b487-7221a42ee4db\" (UID: \"86101de1-ddcb-4d84-b487-7221a42ee4db\") "
Oct 30 20:11:48.040997 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.040975    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86101de1-ddcb-4d84-b487-7221a42ee4db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86101de1-ddcb-4d84-b487-7221a42ee4db-maas-api-xks-ca\") pod \"86101de1-ddcb-4d84-b487-7221a42ee4db\" (UID: \"86101de1-ddcb-4d84-b487-7221a42ee4db\") "
Oct 30 20:11:48.041267 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.041250    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-maas-api-xks-ca\") pod \"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d\" (UID: \"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d\") "
Oct 30 20:11:48.041349 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.041322    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86101de1-ddcb-4d84-b487-7221a42ee4db-maas-api-openshift-ca" pod "86101de1-ddcb-4d84-b487-7221a42ee4db" (UID: "86101de1-ddcb-4d84-b487-7221a42ee4db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:48.041416 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.041330    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86101de1-ddcb-4d84-b487-7221a42ee4db-kube-api-access-ff4qv\" (UniqueName: \"kubernetes.io/projected/86101de1-ddcb-4d84-b487-7221a42ee4db-kube-api-access-ff4qv\") pod \"86101de1-ddcb-4d84-b487-7221a42ee4db\" (UID: \"86101de1-ddcb-4d84-b487-7221a42ee4db\") "
Oct 30 20:11:48.041485 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.041428    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-maas-api-openshift-ca\") pod \"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d\" (UID: \"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d\") "
Oct 30 20:11:48.041972 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.041944    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-maas-api-openshift-ca" pod "c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d" (UID: "c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:48.042604 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.042584    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:48.042726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.042611    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86101de1-ddcb-4d84-b487-7221a42ee4db-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:48.043576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.043548    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86101de1-ddcb-4d84-b487-7221a42ee4db-kube-api-access-ff4qv" pod "86101de1-ddcb-4d84-b487-7221a42ee4db" (UID: "86101de1-ddcb-4d84-b487-7221a42ee4db"). InnerVolumeSpecName "kube-api-access-ff4qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:48.043674 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.043586    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-maas-api-xks-ca" pod "c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d" (UID: "c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:48.043777 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.043751    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86101de1-ddcb-4d84-b487-7221a42ee4db-maas-api-xks-ca" pod "86101de1-ddcb-4d84-b487-7221a42ee4db" (UID: "86101de1-ddcb-4d84-b487-7221a42ee4db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:48.043931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.043780    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-kube-api-access-5r2fn" pod "c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d" (UID: "c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d"). InnerVolumeSpecName "kube-api-access-5r2fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:48.144042 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.144010    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r2fn\" (UniqueName: \"kubernetes.io/projected/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-kube-api-access-5r2fn\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:48.144042 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.144040    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86101de1-ddcb-4d84-b487-7221a42ee4db-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:48.144262 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.144052    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:48.144262 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.144061    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff4qv\" (UniqueName: \"kubernetes.io/projected/86101de1-ddcb-4d84-b487-7221a42ee4db-kube-api-access-ff4qv\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:48.688235 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.688205    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"
Oct 30 20:11:48.688687 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.688211    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p" event={"ID":"86101de1-ddcb-4d84-b487-7221a42ee4db","Type":"ContainerDied","Data":"0448f81451eb5882a32d0f54b9922df7d0313e202b8a62a699124f7c542287fb"}
Oct 30 20:11:48.688687 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.688346    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0448f81451eb5882a32d0f54b9922df7d0313e202b8a62a699124f7c542287fb"
Oct 30 20:11:48.689959 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.689933    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"
Oct 30 20:11:48.690073 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.689979    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f" event={"ID":"c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d","Type":"ContainerDied","Data":"8d2516139751f5f5b0097bcdc94bc926feb19b845cae41e6b7d5102fdcf56ebb"}
Oct 30 20:11:48.690073 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.690018    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d2516139751f5f5b0097bcdc94bc926feb19b845cae41e6b7d5102fdcf56ebb"
Oct 30 20:11:48.692454 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.692425    4823 generic.go:356] "Generic (PLEG): container finished" podID="1aed90ad-aaaf-418d-982d-58ee684f1514" containerID="d014794c049f4b152d9c8aef77e87ddb60bdbbd8091e9db149b50638bcf8bf34" exitCode=0
Oct 30 20:11:48.694362 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:48.693302    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq" event={"ID":"1aed90ad-aaaf-418d-982d-58ee684f1514","Type":"ContainerDied","Data":"d014794c049f4b152d9c8aef77e87ddb60bdbbd8091e9db149b50638bcf8bf34"}
Oct 30 20:11:50.101899 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:50.101489    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"
Oct 30 20:11:50.213773 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:50.213738    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aed90ad-aaaf-418d-982d-58ee684f1514-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aed90ad-aaaf-418d-982d-58ee684f1514-maas-api-openshift-ca\") pod \"1aed90ad-aaaf-418d-982d-58ee684f1514\" (UID: \"1aed90ad-aaaf-418d-982d-58ee684f1514\") "
Oct 30 20:11:50.214031 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:50.213815    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aed90ad-aaaf-418d-982d-58ee684f1514-kube-api-access-4pzmj\" (UniqueName: \"kubernetes.io/projected/1aed90ad-aaaf-418d-982d-58ee684f1514-kube-api-access-4pzmj\") pod \"1aed90ad-aaaf-418d-982d-58ee684f1514\" (UID: \"1aed90ad-aaaf-418d-982d-58ee684f1514\") "
Oct 30 20:11:50.214031 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:50.213934    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aed90ad-aaaf-418d-982d-58ee684f1514-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aed90ad-aaaf-418d-982d-58ee684f1514-maas-api-xks-ca\") pod \"1aed90ad-aaaf-418d-982d-58ee684f1514\" (UID: \"1aed90ad-aaaf-418d-982d-58ee684f1514\") "
Oct 30 20:11:50.214198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:50.214159    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aed90ad-aaaf-418d-982d-58ee684f1514-maas-api-openshift-ca" pod "1aed90ad-aaaf-418d-982d-58ee684f1514" (UID: "1aed90ad-aaaf-418d-982d-58ee684f1514"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:50.214558 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:50.214536    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aed90ad-aaaf-418d-982d-58ee684f1514-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:50.216212 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:50.216183    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aed90ad-aaaf-418d-982d-58ee684f1514-maas-api-xks-ca" pod "1aed90ad-aaaf-418d-982d-58ee684f1514" (UID: "1aed90ad-aaaf-418d-982d-58ee684f1514"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:50.216302 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:50.216249    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aed90ad-aaaf-418d-982d-58ee684f1514-kube-api-access-4pzmj" pod "1aed90ad-aaaf-418d-982d-58ee684f1514" (UID: "1aed90ad-aaaf-418d-982d-58ee684f1514"). InnerVolumeSpecName "kube-api-access-4pzmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:50.315573 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:50.315502    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aed90ad-aaaf-418d-982d-58ee684f1514-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:50.315573 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:50.315533    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pzmj\" (UniqueName: \"kubernetes.io/projected/1aed90ad-aaaf-418d-982d-58ee684f1514-kube-api-access-4pzmj\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:50.716041 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:50.716008    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"
Oct 30 20:11:50.716240 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:50.716036    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq" event={"ID":"1aed90ad-aaaf-418d-982d-58ee684f1514","Type":"ContainerDied","Data":"e2f757e953b232e3062f99d4d73c22c579be12c2aaf7e209d639b90e500035bc"}
Oct 30 20:11:50.716240 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:50.716096    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2f757e953b232e3062f99d4d73c22c579be12c2aaf7e209d639b90e500035bc"
Oct 30 20:11:51.570497 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.570469    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:11:51.751557 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.751465    4823 generic.go:356] "Generic (PLEG): container finished" podID="04c51f58-c2b5-4cef-858e-512b42dba670" containerID="0d9aab1ff4d48482600f093963ef51da5b0d9e6dcf8cdc39962226d28cc60033" exitCode=0
Oct 30 20:11:51.751557 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.751514    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"
Oct 30 20:11:51.751557 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.751524    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq" event={"ID":"04c51f58-c2b5-4cef-858e-512b42dba670","Type":"ContainerDied","Data":"0d9aab1ff4d48482600f093963ef51da5b0d9e6dcf8cdc39962226d28cc60033"}
Oct 30 20:11:51.751850 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.751571    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq" event={"ID":"04c51f58-c2b5-4cef-858e-512b42dba670","Type":"ContainerDied","Data":"a06a0b85163faa0f3f8596d6386b0485f68f8f35793c146e2673a3c9340da913"}
Oct 30 20:11:51.751850 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.751597    4823 scope.go:122] "RemoveContainer" containerID="0d9aab1ff4d48482600f093963ef51da5b0d9e6dcf8cdc39962226d28cc60033"
Oct 30 20:11:51.756229 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.756205    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04c51f58-c2b5-4cef-858e-512b42dba670-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c51f58-c2b5-4cef-858e-512b42dba670-istiod-ca-cert\") pod \"04c51f58-c2b5-4cef-858e-512b42dba670\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") "
Oct 30 20:11:51.756345 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.756322    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-istio-envoy\") pod \"04c51f58-c2b5-4cef-858e-512b42dba670\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") "
Oct 30 20:11:51.756397 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.756379    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-istio-data\") pod \"04c51f58-c2b5-4cef-858e-512b42dba670\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") "
Oct 30 20:11:51.756490 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.756472    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-workload-socket\") pod \"04c51f58-c2b5-4cef-858e-512b42dba670\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") "
Oct 30 20:11:51.756541 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.756517    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-workload-certs\") pod \"04c51f58-c2b5-4cef-858e-512b42dba670\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") "
Oct 30 20:11:51.756589 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.756575    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c51f58-c2b5-4cef-858e-512b42dba670-istio-token\" (UniqueName: \"kubernetes.io/projected/04c51f58-c2b5-4cef-858e-512b42dba670-istio-token\") pod \"04c51f58-c2b5-4cef-858e-512b42dba670\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") "
Oct 30 20:11:51.756753 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.756729    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c51f58-c2b5-4cef-858e-512b42dba670-istiod-ca-cert" pod "04c51f58-c2b5-4cef-858e-512b42dba670" (UID: "04c51f58-c2b5-4cef-858e-512b42dba670"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:51.756841 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.756784    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04c51f58-c2b5-4cef-858e-512b42dba670-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c51f58-c2b5-4cef-858e-512b42dba670-istio-podinfo\") pod \"04c51f58-c2b5-4cef-858e-512b42dba670\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") "
Oct 30 20:11:51.756841 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.756808    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-workload-socket" pod "04c51f58-c2b5-4cef-858e-512b42dba670" (UID: "04c51f58-c2b5-4cef-858e-512b42dba670"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:51.756841 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.756830    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-workload-certs" pod "04c51f58-c2b5-4cef-858e-512b42dba670" (UID: "04c51f58-c2b5-4cef-858e-512b42dba670"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:51.757032 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.756834    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c51f58-c2b5-4cef-858e-512b42dba670-kube-api-access-p6fs7\" (UniqueName: \"kubernetes.io/projected/04c51f58-c2b5-4cef-858e-512b42dba670-kube-api-access-p6fs7\") pod \"04c51f58-c2b5-4cef-858e-512b42dba670\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") "
Oct 30 20:11:51.757032 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.756933    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-credential-socket\") pod \"04c51f58-c2b5-4cef-858e-512b42dba670\" (UID: \"04c51f58-c2b5-4cef-858e-512b42dba670\") "
Oct 30 20:11:51.757256 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.757234    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-credential-socket" pod "04c51f58-c2b5-4cef-858e-512b42dba670" (UID: "04c51f58-c2b5-4cef-858e-512b42dba670"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:51.757915 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.757849    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:51.757915 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.757916    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c51f58-c2b5-4cef-858e-512b42dba670-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:51.758066 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.757926    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:51.758066 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.757938    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:51.758066 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.758003    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-istio-data" pod "04c51f58-c2b5-4cef-858e-512b42dba670" (UID: "04c51f58-c2b5-4cef-858e-512b42dba670"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:51.759192 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.759165    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-istio-envoy" pod "04c51f58-c2b5-4cef-858e-512b42dba670" (UID: "04c51f58-c2b5-4cef-858e-512b42dba670"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:51.759488 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.759465    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04c51f58-c2b5-4cef-858e-512b42dba670-istio-podinfo" pod "04c51f58-c2b5-4cef-858e-512b42dba670" (UID: "04c51f58-c2b5-4cef-858e-512b42dba670"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:11:51.759592 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.759576    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c51f58-c2b5-4cef-858e-512b42dba670-istio-token" pod "04c51f58-c2b5-4cef-858e-512b42dba670" (UID: "04c51f58-c2b5-4cef-858e-512b42dba670"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:51.759637 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.759593    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c51f58-c2b5-4cef-858e-512b42dba670-kube-api-access-p6fs7" pod "04c51f58-c2b5-4cef-858e-512b42dba670" (UID: "04c51f58-c2b5-4cef-858e-512b42dba670"). InnerVolumeSpecName "kube-api-access-p6fs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:51.766755 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.766733    4823 scope.go:122] "RemoveContainer" containerID="170d06bcbe3fa01a2f5ba04610511898c74a189174ec323cbbfb5e15f5f3eb49"
Oct 30 20:11:51.805042 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.805016    4823 scope.go:122] "RemoveContainer" containerID="0d9aab1ff4d48482600f093963ef51da5b0d9e6dcf8cdc39962226d28cc60033"
Oct 30 20:11:51.805421 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:11:51.805388    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9aab1ff4d48482600f093963ef51da5b0d9e6dcf8cdc39962226d28cc60033\": container with ID starting with 0d9aab1ff4d48482600f093963ef51da5b0d9e6dcf8cdc39962226d28cc60033 not found: ID does not exist" containerID="0d9aab1ff4d48482600f093963ef51da5b0d9e6dcf8cdc39962226d28cc60033"
Oct 30 20:11:51.805519 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.805420    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9aab1ff4d48482600f093963ef51da5b0d9e6dcf8cdc39962226d28cc60033"} err="failed to get container status \"0d9aab1ff4d48482600f093963ef51da5b0d9e6dcf8cdc39962226d28cc60033\": rpc error: code = NotFound desc = could not find container \"0d9aab1ff4d48482600f093963ef51da5b0d9e6dcf8cdc39962226d28cc60033\": container with ID starting with 0d9aab1ff4d48482600f093963ef51da5b0d9e6dcf8cdc39962226d28cc60033 not found: ID does not exist"
Oct 30 20:11:51.805519 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.805444    4823 scope.go:122] "RemoveContainer" containerID="170d06bcbe3fa01a2f5ba04610511898c74a189174ec323cbbfb5e15f5f3eb49"
Oct 30 20:11:51.805813 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:11:51.805786    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"170d06bcbe3fa01a2f5ba04610511898c74a189174ec323cbbfb5e15f5f3eb49\": container with ID starting with 170d06bcbe3fa01a2f5ba04610511898c74a189174ec323cbbfb5e15f5f3eb49 not found: ID does not exist" containerID="170d06bcbe3fa01a2f5ba04610511898c74a189174ec323cbbfb5e15f5f3eb49"
Oct 30 20:11:51.805909 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.805825    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"170d06bcbe3fa01a2f5ba04610511898c74a189174ec323cbbfb5e15f5f3eb49"} err="failed to get container status \"170d06bcbe3fa01a2f5ba04610511898c74a189174ec323cbbfb5e15f5f3eb49\": rpc error: code = NotFound desc = could not find container \"170d06bcbe3fa01a2f5ba04610511898c74a189174ec323cbbfb5e15f5f3eb49\": container with ID starting with 170d06bcbe3fa01a2f5ba04610511898c74a189174ec323cbbfb5e15f5f3eb49 not found: ID does not exist"
Oct 30 20:11:51.860483 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.860329    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c51f58-c2b5-4cef-858e-512b42dba670-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:51.860483 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.860361    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c51f58-c2b5-4cef-858e-512b42dba670-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:51.860483 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.860379    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6fs7\" (UniqueName: \"kubernetes.io/projected/04c51f58-c2b5-4cef-858e-512b42dba670-kube-api-access-p6fs7\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:51.860483 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.860393    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:51.860483 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:51.860409    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c51f58-c2b5-4cef-858e-512b42dba670-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:52.030658 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.030558    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"]
Oct 30 20:11:52.031793 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.031753    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n" podUID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" containerName="istio-proxy" containerID="cri-o://d1c4a1a14dd539165306051c8cc55505d2ebcbefb42077eaefc0d04215f3da3e" gracePeriod=30
Oct 30 20:11:52.085288 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.085253    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"]
Oct 30 20:11:52.092273 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.092235    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a7bb6c-openshift-default-656646c7cc-n2zxq"]
Oct 30 20:11:52.328937 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.325096    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04c51f58-c2b5-4cef-858e-512b42dba670" path="/var/lib/kubelet/pods/04c51f58-c2b5-4cef-858e-512b42dba670/volumes"
Oct 30 20:11:52.509051 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.509021    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"]
Oct 30 20:11:52.509788 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.509765    4823 cpu_manager.go:414] "containerMap: removing container" podUID="c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d" containerName="revoke-keys"
Oct 30 20:11:52.509914 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.509790    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d" containerName="revoke-keys"
Oct 30 20:11:52.509914 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.509840    4823 cpu_manager.go:414] "containerMap: removing container" podUID="04c51f58-c2b5-4cef-858e-512b42dba670" containerName="istio-proxy"
Oct 30 20:11:52.509914 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.509847    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c51f58-c2b5-4cef-858e-512b42dba670" containerName="istio-proxy"
Oct 30 20:11:52.509914 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.509876    4823 cpu_manager.go:414] "containerMap: removing container" podUID="04c51f58-c2b5-4cef-858e-512b42dba670" containerName="istio-proxy"
Oct 30 20:11:52.509914 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.509884    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c51f58-c2b5-4cef-858e-512b42dba670" containerName="istio-proxy"
Oct 30 20:11:52.509914 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.509905    4823 cpu_manager.go:414] "containerMap: removing container" podUID="1aed90ad-aaaf-418d-982d-58ee684f1514" containerName="revoke-keys"
Oct 30 20:11:52.509914 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.509912    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aed90ad-aaaf-418d-982d-58ee684f1514" containerName="revoke-keys"
Oct 30 20:11:52.510264 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.509941    4823 cpu_manager.go:414] "containerMap: removing container" podUID="86101de1-ddcb-4d84-b487-7221a42ee4db" containerName="revoke-keys"
Oct 30 20:11:52.510264 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.509948    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86101de1-ddcb-4d84-b487-7221a42ee4db" containerName="revoke-keys"
Oct 30 20:11:52.510264 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.510131    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aed90ad-aaaf-418d-982d-58ee684f1514" containerName="revoke-keys"
Oct 30 20:11:52.510264 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.510148    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="86101de1-ddcb-4d84-b487-7221a42ee4db" containerName="revoke-keys"
Oct 30 20:11:52.510264 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.510176    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d" containerName="revoke-keys"
Oct 30 20:11:52.510264 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.510194    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c51f58-c2b5-4cef-858e-512b42dba670" containerName="istio-proxy"
Oct 30 20:11:52.510264 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.510203    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c51f58-c2b5-4cef-858e-512b42dba670" containerName="istio-proxy"
Oct 30 20:11:52.515467 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.515447    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"
Oct 30 20:11:52.522844 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.522818    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:11:52.523089 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.522966    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:11:52.530031 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.528304    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"]
Oct 30 20:11:52.588001 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.587923    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c5b746-9543-43b1-b132-ebe7ba633d9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq\" (UID: \"01c5b746-9543-43b1-b132-ebe7ba633d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"
Oct 30 20:11:52.588348 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.588005    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c5b746-9543-43b1-b132-ebe7ba633d9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq\" (UID: \"01c5b746-9543-43b1-b132-ebe7ba633d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"
Oct 30 20:11:52.588348 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.588050    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dbwh\" (UniqueName: \"kubernetes.io/projected/01c5b746-9543-43b1-b132-ebe7ba633d9d-kube-api-access-4dbwh\") pod \"maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq\" (UID: \"01c5b746-9543-43b1-b132-ebe7ba633d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"
Oct 30 20:11:52.689958 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.689923    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c5b746-9543-43b1-b132-ebe7ba633d9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq\" (UID: \"01c5b746-9543-43b1-b132-ebe7ba633d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"
Oct 30 20:11:52.690131 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.690018    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c5b746-9543-43b1-b132-ebe7ba633d9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq\" (UID: \"01c5b746-9543-43b1-b132-ebe7ba633d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"
Oct 30 20:11:52.690131 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.690066    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dbwh\" (UniqueName: \"kubernetes.io/projected/01c5b746-9543-43b1-b132-ebe7ba633d9d-kube-api-access-4dbwh\") pod \"maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq\" (UID: \"01c5b746-9543-43b1-b132-ebe7ba633d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"
Oct 30 20:11:52.690600 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.690581    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c5b746-9543-43b1-b132-ebe7ba633d9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq\" (UID: \"01c5b746-9543-43b1-b132-ebe7ba633d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"
Oct 30 20:11:52.692444 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.692424    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c5b746-9543-43b1-b132-ebe7ba633d9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq\" (UID: \"01c5b746-9543-43b1-b132-ebe7ba633d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"
Oct 30 20:11:52.702076 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.702046    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dbwh\" (UniqueName: \"kubernetes.io/projected/01c5b746-9543-43b1-b132-ebe7ba633d9d-kube-api-access-4dbwh\") pod \"maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq\" (UID: \"01c5b746-9543-43b1-b132-ebe7ba633d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"
Oct 30 20:11:52.833074 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:52.833045    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"
Oct 30 20:11:53.123936 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:53.123853    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"]
Oct 30 20:11:53.125749 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:11:53.125722    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c5b746_9543_43b1_b132_ebe7ba633d9d.slice/crio-65401e37bd648e611caeaf7bbf8063b79a374f1f5a9336eeda49fe6373350ca5 WatchSource:0}: Error finding container 65401e37bd648e611caeaf7bbf8063b79a374f1f5a9336eeda49fe6373350ca5: Status 404 returned error can't find the container with id 65401e37bd648e611caeaf7bbf8063b79a374f1f5a9336eeda49fe6373350ca5
Oct 30 20:11:53.770901 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:53.770783    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq" event={"ID":"01c5b746-9543-43b1-b132-ebe7ba633d9d","Type":"ContainerStarted","Data":"6ef42408679cd75cee44534f7f85ff645ce7b9540012a940706d56be6ad2aa63"}
Oct 30 20:11:53.770901 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:53.770835    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq" event={"ID":"01c5b746-9543-43b1-b132-ebe7ba633d9d","Type":"ContainerStarted","Data":"65401e37bd648e611caeaf7bbf8063b79a374f1f5a9336eeda49fe6373350ca5"}
Oct 30 20:11:53.793794 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:53.793737    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq" podStartSLOduration=1.793723709 podStartE2EDuration="1.793723709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:53.791418123 +0000 UTC m=+2513.684032446" watchObservedRunningTime="2026-10-30 20:11:53.793723709 +0000 UTC m=+2513.686338033"
Oct 30 20:11:53.861465 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:53.861431    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"]
Oct 30 20:11:54.616894 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.616851    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"]
Oct 30 20:11:54.621633 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.621607    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"]
Oct 30 20:11:54.621769 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.621720    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"
Oct 30 20:11:54.626664 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.626629    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"
Oct 30 20:11:54.631150 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.631127    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"]
Oct 30 20:11:54.636004 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.635980    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"]
Oct 30 20:11:54.646777 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.646745    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56935a3f-8vrjn\" (UID: \"db51eeaa-23d4-4ade-9e6e-6c59fae623c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"
Oct 30 20:11:54.647095 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.647073    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56935a3f-8vrjn\" (UID: \"db51eeaa-23d4-4ade-9e6e-6c59fae623c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"
Oct 30 20:11:54.647175 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.647160    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57zrs\" (UniqueName: \"kubernetes.io/projected/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-kube-api-access-57zrs\") pod \"maas-api-revoke-sub-56935a3f-8vrjn\" (UID: \"db51eeaa-23d4-4ade-9e6e-6c59fae623c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"
Oct 30 20:11:54.748737 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.748707    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82225a9f-9fqmh\" (UID: \"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"
Oct 30 20:11:54.748897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.748840    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82225a9f-9fqmh\" (UID: \"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"
Oct 30 20:11:54.748955 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.748893    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njxgk\" (UniqueName: \"kubernetes.io/projected/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-kube-api-access-njxgk\") pod \"maas-api-revoke-sub-82225a9f-9fqmh\" (UID: \"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"
Oct 30 20:11:54.749016 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.748999    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56935a3f-8vrjn\" (UID: \"db51eeaa-23d4-4ade-9e6e-6c59fae623c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"
Oct 30 20:11:54.749086 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.749060    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57zrs\" (UniqueName: \"kubernetes.io/projected/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-kube-api-access-57zrs\") pod \"maas-api-revoke-sub-56935a3f-8vrjn\" (UID: \"db51eeaa-23d4-4ade-9e6e-6c59fae623c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"
Oct 30 20:11:54.749198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.749146    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56935a3f-8vrjn\" (UID: \"db51eeaa-23d4-4ade-9e6e-6c59fae623c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"
Oct 30 20:11:54.749794 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.749772    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56935a3f-8vrjn\" (UID: \"db51eeaa-23d4-4ade-9e6e-6c59fae623c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"
Oct 30 20:11:54.751418 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.751393    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56935a3f-8vrjn\" (UID: \"db51eeaa-23d4-4ade-9e6e-6c59fae623c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"
Oct 30 20:11:54.764437 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.764415    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57zrs\" (UniqueName: \"kubernetes.io/projected/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-kube-api-access-57zrs\") pod \"maas-api-revoke-sub-56935a3f-8vrjn\" (UID: \"db51eeaa-23d4-4ade-9e6e-6c59fae623c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"
Oct 30 20:11:54.781686 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.781657    4823 generic.go:356] "Generic (PLEG): container finished" podID="01c5b746-9543-43b1-b132-ebe7ba633d9d" containerID="6ef42408679cd75cee44534f7f85ff645ce7b9540012a940706d56be6ad2aa63" exitCode=0
Oct 30 20:11:54.781987 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.781738    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq" event={"ID":"01c5b746-9543-43b1-b132-ebe7ba633d9d","Type":"ContainerDied","Data":"6ef42408679cd75cee44534f7f85ff645ce7b9540012a940706d56be6ad2aa63"}
Oct 30 20:11:54.851184 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.851148    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82225a9f-9fqmh\" (UID: \"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"
Oct 30 20:11:54.851397 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.851217    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njxgk\" (UniqueName: \"kubernetes.io/projected/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-kube-api-access-njxgk\") pod \"maas-api-revoke-sub-82225a9f-9fqmh\" (UID: \"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"
Oct 30 20:11:54.851512 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.851495    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82225a9f-9fqmh\" (UID: \"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"
Oct 30 20:11:54.852615 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.851991    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82225a9f-9fqmh\" (UID: \"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"
Oct 30 20:11:54.854675 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.854554    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82225a9f-9fqmh\" (UID: \"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"
Oct 30 20:11:54.868632 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.868557    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njxgk\" (UniqueName: \"kubernetes.io/projected/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-kube-api-access-njxgk\") pod \"maas-api-revoke-sub-82225a9f-9fqmh\" (UID: \"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"
Oct 30 20:11:54.941416 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.941388    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"
Oct 30 20:11:54.951673 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:54.951646    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"
Oct 30 20:11:55.347931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:55.338505    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"]
Oct 30 20:11:55.456384 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:55.456354    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"]
Oct 30 20:11:55.457506 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:11:55.457481    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6e7d8f_a41b_4609_a6b4_e0642890b4b4.slice/crio-6a130714e947293a8396a66c4f81037bbe9960389a6994ab2a3011908cee0b38 WatchSource:0}: Error finding container 6a130714e947293a8396a66c4f81037bbe9960389a6994ab2a3011908cee0b38: Status 404 returned error can't find the container with id 6a130714e947293a8396a66c4f81037bbe9960389a6994ab2a3011908cee0b38
Oct 30 20:11:55.653993 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:55.653939    4823 patch_prober.go:29] interesting pod/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:11:55.654188 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:55.654011    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n" podUID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:11:55.838082 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:55.838049    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh" event={"ID":"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4","Type":"ContainerStarted","Data":"6a130714e947293a8396a66c4f81037bbe9960389a6994ab2a3011908cee0b38"}
Oct 30 20:11:55.841020 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:55.840990    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn" event={"ID":"db51eeaa-23d4-4ade-9e6e-6c59fae623c0","Type":"ContainerStarted","Data":"fad57e5c23b1b11d12f8a0e59ae1254eada0264722ebe75fce7be969ef7c19bb"}
Oct 30 20:11:56.018555 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.018525    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"
Oct 30 20:11:56.115840 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.115798    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c5b746-9543-43b1-b132-ebe7ba633d9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c5b746-9543-43b1-b132-ebe7ba633d9d-maas-api-openshift-ca\") pod \"01c5b746-9543-43b1-b132-ebe7ba633d9d\" (UID: \"01c5b746-9543-43b1-b132-ebe7ba633d9d\") "
Oct 30 20:11:56.116026 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.115994    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c5b746-9543-43b1-b132-ebe7ba633d9d-kube-api-access-4dbwh\" (UniqueName: \"kubernetes.io/projected/01c5b746-9543-43b1-b132-ebe7ba633d9d-kube-api-access-4dbwh\") pod \"01c5b746-9543-43b1-b132-ebe7ba633d9d\" (UID: \"01c5b746-9543-43b1-b132-ebe7ba633d9d\") "
Oct 30 20:11:56.116155 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.116142    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c5b746-9543-43b1-b132-ebe7ba633d9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c5b746-9543-43b1-b132-ebe7ba633d9d-maas-api-xks-ca\") pod \"01c5b746-9543-43b1-b132-ebe7ba633d9d\" (UID: \"01c5b746-9543-43b1-b132-ebe7ba633d9d\") "
Oct 30 20:11:56.116221 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.116197    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c5b746-9543-43b1-b132-ebe7ba633d9d-maas-api-openshift-ca" pod "01c5b746-9543-43b1-b132-ebe7ba633d9d" (UID: "01c5b746-9543-43b1-b132-ebe7ba633d9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:56.116602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.116585    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c5b746-9543-43b1-b132-ebe7ba633d9d-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:56.118246 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.118223    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c5b746-9543-43b1-b132-ebe7ba633d9d-kube-api-access-4dbwh" pod "01c5b746-9543-43b1-b132-ebe7ba633d9d" (UID: "01c5b746-9543-43b1-b132-ebe7ba633d9d"). InnerVolumeSpecName "kube-api-access-4dbwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:56.118342 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.118321    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c5b746-9543-43b1-b132-ebe7ba633d9d-maas-api-xks-ca" pod "01c5b746-9543-43b1-b132-ebe7ba633d9d" (UID: "01c5b746-9543-43b1-b132-ebe7ba633d9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:56.217837 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.217756    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dbwh\" (UniqueName: \"kubernetes.io/projected/01c5b746-9543-43b1-b132-ebe7ba633d9d-kube-api-access-4dbwh\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:56.217837 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.217787    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c5b746-9543-43b1-b132-ebe7ba633d9d-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:56.906038 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.905977    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"
Oct 30 20:11:56.906690 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.906044    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq" event={"ID":"01c5b746-9543-43b1-b132-ebe7ba633d9d","Type":"ContainerDied","Data":"65401e37bd648e611caeaf7bbf8063b79a374f1f5a9336eeda49fe6373350ca5"}
Oct 30 20:11:56.906690 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.906477    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65401e37bd648e611caeaf7bbf8063b79a374f1f5a9336eeda49fe6373350ca5"
Oct 30 20:11:56.909271 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.909240    4823 generic.go:356] "Generic (PLEG): container finished" podID="3c6e7d8f-a41b-4609-a6b4-e0642890b4b4" containerID="59c661798bab69bb7e291a2abcb48bd684b8155ea9f39ba0f7decdbee8fdc492" exitCode=0
Oct 30 20:11:56.909483 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.909461    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh" event={"ID":"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4","Type":"ContainerDied","Data":"59c661798bab69bb7e291a2abcb48bd684b8155ea9f39ba0f7decdbee8fdc492"}
Oct 30 20:11:56.911375 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.911349    4823 generic.go:356] "Generic (PLEG): container finished" podID="db51eeaa-23d4-4ade-9e6e-6c59fae623c0" containerID="dcfd734d173dd9ef6516c70bbb935af53d62e1bc336e475907dc3a90cccf675f" exitCode=0
Oct 30 20:11:56.911481 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.911443    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn" event={"ID":"db51eeaa-23d4-4ade-9e6e-6c59fae623c0","Type":"ContainerDied","Data":"dcfd734d173dd9ef6516c70bbb935af53d62e1bc336e475907dc3a90cccf675f"}
Oct 30 20:11:56.948385 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.948355    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"]
Oct 30 20:11:56.975109 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:56.975077    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f244d-99qj5"]
Oct 30 20:11:57.400309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.400284    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:11:57.497444 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.497361    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-token\" (UniqueName: \"kubernetes.io/projected/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-token\") pod \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") "
Oct 30 20:11:57.497619 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.497471    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-podinfo\") pod \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") "
Oct 30 20:11:57.497619 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.497541    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-workload-certs\") pod \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") "
Oct 30 20:11:57.497619 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.497590    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-credential-socket\") pod \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") "
Oct 30 20:11:57.497828 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.497804    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-data\") pod \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") "
Oct 30 20:11:57.497928 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.497915    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-envoy\") pod \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") "
Oct 30 20:11:57.497986 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.497973    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istiod-ca-cert\") pod \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") "
Oct 30 20:11:57.498040 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.498019    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-workload-socket\") pod \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") "
Oct 30 20:11:57.498095 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.498058    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-kube-api-access-j7f2h\" (UniqueName: \"kubernetes.io/projected/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-kube-api-access-j7f2h\") pod \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\" (UID: \"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3\") "
Oct 30 20:11:57.500240 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.498010    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-workload-certs" pod "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" (UID: "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:57.500600 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.500567    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-envoy" pod "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" (UID: "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:57.500703 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.500632    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istiod-ca-cert" pod "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" (UID: "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:57.501112 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.500952    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-token" pod "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" (UID: "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:57.501112 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.500965    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-credential-socket" pod "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" (UID: "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:57.501112 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.498943    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-data" pod "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" (UID: "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:57.501294 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.501206    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-workload-socket" pod "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" (UID: "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:11:57.502066 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.502037    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-podinfo" pod "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" (UID: "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:11:57.502327 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.502311    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-kube-api-access-j7f2h" pod "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" (UID: "9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3"). InnerVolumeSpecName "kube-api-access-j7f2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:57.600927 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.600882    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:57.600927 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.600911    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:57.600927 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.600921    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:57.600927 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.600930    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:57.600927 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.600939    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:57.601222 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.600948    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:57.601222 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.600956    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:57.601222 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.600966    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7f2h\" (UniqueName: \"kubernetes.io/projected/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-kube-api-access-j7f2h\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:57.601222 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.600975    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:57.963568 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.963525    4823 generic.go:356] "Generic (PLEG): container finished" podID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" containerID="d1c4a1a14dd539165306051c8cc55505d2ebcbefb42077eaefc0d04215f3da3e" exitCode=0
Oct 30 20:11:57.964262 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.964134    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"
Oct 30 20:11:57.965032 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.964997    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n" event={"ID":"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3","Type":"ContainerDied","Data":"d1c4a1a14dd539165306051c8cc55505d2ebcbefb42077eaefc0d04215f3da3e"}
Oct 30 20:11:57.965157 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.965060    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n" event={"ID":"9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3","Type":"ContainerDied","Data":"79509709aa674380892d2d467c7ac2d1d2a893a936a003a485b2bce15df3b85a"}
Oct 30 20:11:57.965157 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:57.965089    4823 scope.go:122] "RemoveContainer" containerID="d1c4a1a14dd539165306051c8cc55505d2ebcbefb42077eaefc0d04215f3da3e"
Oct 30 20:11:58.003100 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.002676    4823 scope.go:122] "RemoveContainer" containerID="895393ad8bd262e70e9d798e3795bb459ae01e232eee17b1cc38488951747dcb"
Oct 30 20:11:58.024107 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.023784    4823 scope.go:122] "RemoveContainer" containerID="d1c4a1a14dd539165306051c8cc55505d2ebcbefb42077eaefc0d04215f3da3e"
Oct 30 20:11:58.024205 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:11:58.024159    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c4a1a14dd539165306051c8cc55505d2ebcbefb42077eaefc0d04215f3da3e\": container with ID starting with d1c4a1a14dd539165306051c8cc55505d2ebcbefb42077eaefc0d04215f3da3e not found: ID does not exist" containerID="d1c4a1a14dd539165306051c8cc55505d2ebcbefb42077eaefc0d04215f3da3e"
Oct 30 20:11:58.024270 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.024203    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c4a1a14dd539165306051c8cc55505d2ebcbefb42077eaefc0d04215f3da3e"} err="failed to get container status \"d1c4a1a14dd539165306051c8cc55505d2ebcbefb42077eaefc0d04215f3da3e\": rpc error: code = NotFound desc = could not find container \"d1c4a1a14dd539165306051c8cc55505d2ebcbefb42077eaefc0d04215f3da3e\": container with ID starting with d1c4a1a14dd539165306051c8cc55505d2ebcbefb42077eaefc0d04215f3da3e not found: ID does not exist"
Oct 30 20:11:58.024270 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.024222    4823 scope.go:122] "RemoveContainer" containerID="895393ad8bd262e70e9d798e3795bb459ae01e232eee17b1cc38488951747dcb"
Oct 30 20:11:58.024553 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:11:58.024505    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"895393ad8bd262e70e9d798e3795bb459ae01e232eee17b1cc38488951747dcb\": container with ID starting with 895393ad8bd262e70e9d798e3795bb459ae01e232eee17b1cc38488951747dcb not found: ID does not exist" containerID="895393ad8bd262e70e9d798e3795bb459ae01e232eee17b1cc38488951747dcb"
Oct 30 20:11:58.024553 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.024538    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"895393ad8bd262e70e9d798e3795bb459ae01e232eee17b1cc38488951747dcb"} err="failed to get container status \"895393ad8bd262e70e9d798e3795bb459ae01e232eee17b1cc38488951747dcb\": rpc error: code = NotFound desc = could not find container \"895393ad8bd262e70e9d798e3795bb459ae01e232eee17b1cc38488951747dcb\": container with ID starting with 895393ad8bd262e70e9d798e3795bb459ae01e232eee17b1cc38488951747dcb not found: ID does not exist"
Oct 30 20:11:58.041692 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.040673    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"]
Oct 30 20:11:58.046284 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.046252    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3a4f02-openshift-default-59bcb57895-h9p7n"]
Oct 30 20:11:58.182400 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.182373    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"
Oct 30 20:11:58.225926 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.225845    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"
Oct 30 20:11:58.281096 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.281061    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f" path="/var/lib/kubelet/pods/2c459648-15c9-4a2f-bb8c-0d7eb3b7ef3f/volumes"
Oct 30 20:11:58.281732 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.281714    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" path="/var/lib/kubelet/pods/9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3/volumes"
Oct 30 20:11:58.323667 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.323633    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-maas-api-xks-ca\") pod \"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4\" (UID: \"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4\") "
Oct 30 20:11:58.323855 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.323786    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-maas-api-openshift-ca\") pod \"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4\" (UID: \"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4\") "
Oct 30 20:11:58.323963 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.323885    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-kube-api-access-57zrs\" (UniqueName: \"kubernetes.io/projected/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-kube-api-access-57zrs\") pod \"db51eeaa-23d4-4ade-9e6e-6c59fae623c0\" (UID: \"db51eeaa-23d4-4ade-9e6e-6c59fae623c0\") "
Oct 30 20:11:58.324026 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.324000    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-maas-api-xks-ca\") pod \"db51eeaa-23d4-4ade-9e6e-6c59fae623c0\" (UID: \"db51eeaa-23d4-4ade-9e6e-6c59fae623c0\") "
Oct 30 20:11:58.324140 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.324124    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-kube-api-access-njxgk\" (UniqueName: \"kubernetes.io/projected/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-kube-api-access-njxgk\") pod \"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4\" (UID: \"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4\") "
Oct 30 20:11:58.324221 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.324166    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-maas-api-openshift-ca\") pod \"db51eeaa-23d4-4ade-9e6e-6c59fae623c0\" (UID: \"db51eeaa-23d4-4ade-9e6e-6c59fae623c0\") "
Oct 30 20:11:58.325886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.325841    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-maas-api-openshift-ca" pod "3c6e7d8f-a41b-4609-a6b4-e0642890b4b4" (UID: "3c6e7d8f-a41b-4609-a6b4-e0642890b4b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:58.326391 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.326361    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-maas-api-openshift-ca" pod "db51eeaa-23d4-4ade-9e6e-6c59fae623c0" (UID: "db51eeaa-23d4-4ade-9e6e-6c59fae623c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:58.327506 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.327478    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-maas-api-xks-ca" pod "db51eeaa-23d4-4ade-9e6e-6c59fae623c0" (UID: "db51eeaa-23d4-4ade-9e6e-6c59fae623c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:58.328913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.327719    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-kube-api-access-57zrs" pod "db51eeaa-23d4-4ade-9e6e-6c59fae623c0" (UID: "db51eeaa-23d4-4ade-9e6e-6c59fae623c0"). InnerVolumeSpecName "kube-api-access-57zrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:58.328913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.327997    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-kube-api-access-njxgk" pod "3c6e7d8f-a41b-4609-a6b4-e0642890b4b4" (UID: "3c6e7d8f-a41b-4609-a6b4-e0642890b4b4"). InnerVolumeSpecName "kube-api-access-njxgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:58.328913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.328672    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-maas-api-xks-ca" pod "3c6e7d8f-a41b-4609-a6b4-e0642890b4b4" (UID: "3c6e7d8f-a41b-4609-a6b4-e0642890b4b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:58.426525 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.426465    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:58.426525 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.426495    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njxgk\" (UniqueName: \"kubernetes.io/projected/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-kube-api-access-njxgk\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:58.426525 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.426506    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:58.426525 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.426519    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:58.426525 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.426533    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:58.426945 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.426542    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57zrs\" (UniqueName: \"kubernetes.io/projected/db51eeaa-23d4-4ade-9e6e-6c59fae623c0-kube-api-access-57zrs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:58.862315 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.862277    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm" podUID="fd0d4ac0-27c1-4735-b333-63b4e75afe70" containerName="payload-processing" containerID="cri-o://635dad2452b6b4b06f14eb51e27df8beae0404ddccb230204de15dff6d17c1b4" gracePeriod=25
Oct 30 20:11:58.890695 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.890658    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"]
Oct 30 20:11:58.894763 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.894735    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d5e3e5-ghfdf"]
Oct 30 20:11:58.975478 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.975447    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"
Oct 30 20:11:58.975931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.975486    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn" event={"ID":"db51eeaa-23d4-4ade-9e6e-6c59fae623c0","Type":"ContainerDied","Data":"fad57e5c23b1b11d12f8a0e59ae1254eada0264722ebe75fce7be969ef7c19bb"}
Oct 30 20:11:58.975931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.975537    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fad57e5c23b1b11d12f8a0e59ae1254eada0264722ebe75fce7be969ef7c19bb"
Oct 30 20:11:58.977122 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.977104    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"
Oct 30 20:11:58.977303 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.977144    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh" event={"ID":"3c6e7d8f-a41b-4609-a6b4-e0642890b4b4","Type":"ContainerDied","Data":"6a130714e947293a8396a66c4f81037bbe9960389a6994ab2a3011908cee0b38"}
Oct 30 20:11:58.977303 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:11:58.977194    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a130714e947293a8396a66c4f81037bbe9960389a6994ab2a3011908cee0b38"
Oct 30 20:12:00.301478 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:00.301426    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d91f3d0-a115-4e98-a683-a8fd017d89d9" path="/var/lib/kubelet/pods/3d91f3d0-a115-4e98-a683-a8fd017d89d9/volumes"
Oct 30 20:12:02.016577 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:12:02.016535    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:12:02.424763 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.424733    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"]
Oct 30 20:12:02.425061 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.424988    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb" podUID="adb403b1-bc1d-4f49-88e1-fbb1baceae02" containerName="maas-api" containerID="cri-o://711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3" gracePeriod=30
Oct 30 20:12:02.746128 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.746101    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:12:02.828803 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.828766    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb403b1-bc1d-4f49-88e1-fbb1baceae02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb403b1-bc1d-4f49-88e1-fbb1baceae02-openshift-service-ca\") pod \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\" (UID: \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\") "
Oct 30 20:12:02.829006 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.828924    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adb403b1-bc1d-4f49-88e1-fbb1baceae02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb403b1-bc1d-4f49-88e1-fbb1baceae02-metrics-certs\") pod \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\" (UID: \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\") "
Oct 30 20:12:02.829076 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.829006    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb403b1-bc1d-4f49-88e1-fbb1baceae02-kube-api-access-ftzk7\" (UniqueName: \"kubernetes.io/projected/adb403b1-bc1d-4f49-88e1-fbb1baceae02-kube-api-access-ftzk7\") pod \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\" (UID: \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\") "
Oct 30 20:12:02.829133 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.829071    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adb403b1-bc1d-4f49-88e1-fbb1baceae02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb403b1-bc1d-4f49-88e1-fbb1baceae02-maas-api-tls\") pod \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\" (UID: \"adb403b1-bc1d-4f49-88e1-fbb1baceae02\") "
Oct 30 20:12:02.829244 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.829216    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb403b1-bc1d-4f49-88e1-fbb1baceae02-openshift-service-ca" pod "adb403b1-bc1d-4f49-88e1-fbb1baceae02" (UID: "adb403b1-bc1d-4f49-88e1-fbb1baceae02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:02.830026 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.829995    4823 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb403b1-bc1d-4f49-88e1-fbb1baceae02-openshift-service-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:02.832439 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.832412    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb403b1-bc1d-4f49-88e1-fbb1baceae02-metrics-certs" pod "adb403b1-bc1d-4f49-88e1-fbb1baceae02" (UID: "adb403b1-bc1d-4f49-88e1-fbb1baceae02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:02.832573 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.832463    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb403b1-bc1d-4f49-88e1-fbb1baceae02-maas-api-tls" pod "adb403b1-bc1d-4f49-88e1-fbb1baceae02" (UID: "adb403b1-bc1d-4f49-88e1-fbb1baceae02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:02.832639 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.832620    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb403b1-bc1d-4f49-88e1-fbb1baceae02-kube-api-access-ftzk7" pod "adb403b1-bc1d-4f49-88e1-fbb1baceae02" (UID: "adb403b1-bc1d-4f49-88e1-fbb1baceae02"). InnerVolumeSpecName "kube-api-access-ftzk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:02.842809 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.842786    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"]
Oct 30 20:12:02.852319 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.852299    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7a733-8frqq"]
Oct 30 20:12:02.931046 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.931016    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftzk7\" (UniqueName: \"kubernetes.io/projected/adb403b1-bc1d-4f49-88e1-fbb1baceae02-kube-api-access-ftzk7\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:02.931046 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.931044    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adb403b1-bc1d-4f49-88e1-fbb1baceae02-maas-api-tls\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:02.931245 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:02.931055    4823 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adb403b1-bc1d-4f49-88e1-fbb1baceae02-metrics-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.018759 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.018558    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main" containerID="cri-o://966de27847ce9e4d0f3da21d8abca77d96a8d6d3c569c23b7625693b17d2f22c" gracePeriod=45
Oct 30 20:12:03.082917 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.071705    4823 generic.go:356] "Generic (PLEG): container finished" podID="adb403b1-bc1d-4f49-88e1-fbb1baceae02" containerID="711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3" exitCode=0
Oct 30 20:12:03.082917 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.071841    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb" event={"ID":"adb403b1-bc1d-4f49-88e1-fbb1baceae02","Type":"ContainerDied","Data":"711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3"}
Oct 30 20:12:03.082917 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.071966    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb" event={"ID":"adb403b1-bc1d-4f49-88e1-fbb1baceae02","Type":"ContainerDied","Data":"526aca620a9ffa6d7d009db2f351be29fdd003bec38061f0e7cedefc314b0459"}
Oct 30 20:12:03.082917 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.072005    4823 scope.go:122] "RemoveContainer" containerID="711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3"
Oct 30 20:12:03.082917 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.072247    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"
Oct 30 20:12:03.183728 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.183679    4823 scope.go:122] "RemoveContainer" containerID="711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3"
Oct 30 20:12:03.195817 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:12:03.195780    4823 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_maas-api_maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb_odh-ai-gateway-infra_adb403b1-bc1d-4f49-88e1-fbb1baceae02_0 in pod sandbox 526aca620a9ffa6d7d009db2f351be29fdd003bec38061f0e7cedefc314b0459 from index: no such id: '711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3'" containerID="711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3"
Oct 30 20:12:03.196137 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.195834    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3"} err="rpc error: code = Unknown desc = failed to delete container k8s_maas-api_maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb_odh-ai-gateway-infra_adb403b1-bc1d-4f49-88e1-fbb1baceae02_0 in pod sandbox 526aca620a9ffa6d7d009db2f351be29fdd003bec38061f0e7cedefc314b0459 from index: no such id: '711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3'"
Oct 30 20:12:03.196137 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.195876    4823 scope.go:122] "RemoveContainer" containerID="711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3"
Oct 30 20:12:03.200702 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:12:03.200656    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3\": container with ID starting with 711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3 not found: ID does not exist" containerID="711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3"
Oct 30 20:12:03.200977 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.200944    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3"} err="failed to get container status \"711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3\": rpc error: code = NotFound desc = could not find container \"711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3\": container with ID starting with 711c2e50bc50c435e587f23b7bbc45a4fb40fa5b3354f16efcac8bfee602b9a3 not found: ID does not exist"
Oct 30 20:12:03.224804 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.224773    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"]
Oct 30 20:12:03.231286 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.231255    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7a733-58cfbdf5d8-pxbgb"]
Oct 30 20:12:03.386024 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.385999    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 20:12:03.442119 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.442085    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-tmp-dir\") pod \"dd230496-5fbe-4500-965c-16e48f099552\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") "
Oct 30 20:12:03.442318 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.442182    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-dshm\") pod \"dd230496-5fbe-4500-965c-16e48f099552\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") "
Oct 30 20:12:03.442318 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.442227    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd230496-5fbe-4500-965c-16e48f099552-kube-api-access-kxqw7\" (UniqueName: \"kubernetes.io/projected/dd230496-5fbe-4500-965c-16e48f099552-kube-api-access-kxqw7\") pod \"dd230496-5fbe-4500-965c-16e48f099552\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") "
Oct 30 20:12:03.442318 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.442296    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-model-cache\") pod \"dd230496-5fbe-4500-965c-16e48f099552\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") "
Oct 30 20:12:03.442518 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.442433    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd230496-5fbe-4500-965c-16e48f099552-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd230496-5fbe-4500-965c-16e48f099552-tls-certs\") pod \"dd230496-5fbe-4500-965c-16e48f099552\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") "
Oct 30 20:12:03.442518 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.442481    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-home\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-home\") pod \"dd230496-5fbe-4500-965c-16e48f099552\" (UID: \"dd230496-5fbe-4500-965c-16e48f099552\") "
Oct 30 20:12:03.443282 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.443236    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-model-cache" pod "dd230496-5fbe-4500-965c-16e48f099552" (UID: "dd230496-5fbe-4500-965c-16e48f099552"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:03.443453 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.443420    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-home" pod "dd230496-5fbe-4500-965c-16e48f099552" (UID: "dd230496-5fbe-4500-965c-16e48f099552"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:03.443600 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.443575    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-tmp-dir" pod "dd230496-5fbe-4500-965c-16e48f099552" (UID: "dd230496-5fbe-4500-965c-16e48f099552"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:03.460434 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.448483    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-dshm" pod "dd230496-5fbe-4500-965c-16e48f099552" (UID: "dd230496-5fbe-4500-965c-16e48f099552"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:03.460434 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.450066    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd230496-5fbe-4500-965c-16e48f099552-tls-certs" pod "dd230496-5fbe-4500-965c-16e48f099552" (UID: "dd230496-5fbe-4500-965c-16e48f099552"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:03.460434 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.455448    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd230496-5fbe-4500-965c-16e48f099552-kube-api-access-kxqw7" pod "dd230496-5fbe-4500-965c-16e48f099552" (UID: "dd230496-5fbe-4500-965c-16e48f099552"). InnerVolumeSpecName "kube-api-access-kxqw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:03.544807 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.544719    4823 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-tmp-dir\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.544807 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.544761    4823 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-dshm\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.544807 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.544772    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxqw7\" (UniqueName: \"kubernetes.io/projected/dd230496-5fbe-4500-965c-16e48f099552-kube-api-access-kxqw7\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.544807 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.544782    4823 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-model-cache\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.544807 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.544793    4823 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd230496-5fbe-4500-965c-16e48f099552-tls-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:03.544807 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:03.544801    4823 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd230496-5fbe-4500-965c-16e48f099552-home\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:04.080599 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:04.080568    4823 generic.go:356] "Generic (PLEG): container finished" podID="dd230496-5fbe-4500-965c-16e48f099552" containerID="966de27847ce9e4d0f3da21d8abca77d96a8d6d3c569c23b7625693b17d2f22c" exitCode=0
Oct 30 20:12:04.081118 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:04.080617    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"
Oct 30 20:12:04.081118 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:04.080650    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" event={"ID":"dd230496-5fbe-4500-965c-16e48f099552","Type":"ContainerDied","Data":"966de27847ce9e4d0f3da21d8abca77d96a8d6d3c569c23b7625693b17d2f22c"}
Oct 30 20:12:04.081118 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:04.080690    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk" event={"ID":"dd230496-5fbe-4500-965c-16e48f099552","Type":"ContainerDied","Data":"df3e62e60a4cda7c93f3240e305f8e8aa950113931485146a4e6090fc2426f81"}
Oct 30 20:12:04.081118 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:04.080716    4823 scope.go:122] "RemoveContainer" containerID="966de27847ce9e4d0f3da21d8abca77d96a8d6d3c569c23b7625693b17d2f22c"
Oct 30 20:12:04.091857 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:04.091836    4823 scope.go:122] "RemoveContainer" containerID="966de27847ce9e4d0f3da21d8abca77d96a8d6d3c569c23b7625693b17d2f22c"
Oct 30 20:12:04.092179 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:12:04.092159    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966de27847ce9e4d0f3da21d8abca77d96a8d6d3c569c23b7625693b17d2f22c\": container with ID starting with 966de27847ce9e4d0f3da21d8abca77d96a8d6d3c569c23b7625693b17d2f22c not found: ID does not exist" containerID="966de27847ce9e4d0f3da21d8abca77d96a8d6d3c569c23b7625693b17d2f22c"
Oct 30 20:12:04.092239 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:04.092188    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966de27847ce9e4d0f3da21d8abca77d96a8d6d3c569c23b7625693b17d2f22c"} err="failed to get container status \"966de27847ce9e4d0f3da21d8abca77d96a8d6d3c569c23b7625693b17d2f22c\": rpc error: code = NotFound desc = could not find container \"966de27847ce9e4d0f3da21d8abca77d96a8d6d3c569c23b7625693b17d2f22c\": container with ID starting with 966de27847ce9e4d0f3da21d8abca77d96a8d6d3c569c23b7625693b17d2f22c not found: ID does not exist"
Oct 30 20:12:04.109035 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:04.109012    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"]
Oct 30 20:12:04.112899 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:04.112855    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/simulator-w1-a1d885-kserve-6794f9f86c-ldhqk"]
Oct 30 20:12:04.307553 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:04.302346    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aed90ad-aaaf-418d-982d-58ee684f1514" path="/var/lib/kubelet/pods/1aed90ad-aaaf-418d-982d-58ee684f1514/volumes"
Oct 30 20:12:04.307553 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:04.303112    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb403b1-bc1d-4f49-88e1-fbb1baceae02" path="/var/lib/kubelet/pods/adb403b1-bc1d-4f49-88e1-fbb1baceae02/volumes"
Oct 30 20:12:04.307553 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:04.303911    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd230496-5fbe-4500-965c-16e48f099552" path="/var/lib/kubelet/pods/dd230496-5fbe-4500-965c-16e48f099552/volumes"
Oct 30 20:12:04.497134 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:04.497101    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"]
Oct 30 20:12:04.508489 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:04.508461    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a8de3b-4pc8f"]
Oct 30 20:12:06.104263 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.104233    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"]
Oct 30 20:12:06.109301 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.109274    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-82885f-z7bjq"]
Oct 30 20:12:06.279992 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.279952    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c5b746-9543-43b1-b132-ebe7ba633d9d" path="/var/lib/kubelet/pods/01c5b746-9543-43b1-b132-ebe7ba633d9d/volumes"
Oct 30 20:12:06.280624 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.280599    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d" path="/var/lib/kubelet/pods/c6f51db7-b2e9-464c-9dd4-1e0c9875fd9d/volumes"
Oct 30 20:12:06.762375 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.762330    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"]
Oct 30 20:12:06.763028 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763013    4823 cpu_manager.go:414] "containerMap: removing container" podUID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" containerName="istio-proxy"
Oct 30 20:12:06.763090 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763030    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" containerName="istio-proxy"
Oct 30 20:12:06.763090 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763046    4823 cpu_manager.go:414] "containerMap: removing container" podUID="01c5b746-9543-43b1-b132-ebe7ba633d9d" containerName="revoke-keys"
Oct 30 20:12:06.763090 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763052    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c5b746-9543-43b1-b132-ebe7ba633d9d" containerName="revoke-keys"
Oct 30 20:12:06.763090 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763068    4823 cpu_manager.go:414] "containerMap: removing container" podUID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" containerName="istio-proxy"
Oct 30 20:12:06.763090 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763073    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" containerName="istio-proxy"
Oct 30 20:12:06.763090 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763083    4823 cpu_manager.go:414] "containerMap: removing container" podUID="db51eeaa-23d4-4ade-9e6e-6c59fae623c0" containerName="revoke-keys"
Oct 30 20:12:06.763090 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763088    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db51eeaa-23d4-4ade-9e6e-6c59fae623c0" containerName="revoke-keys"
Oct 30 20:12:06.763323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763121    4823 cpu_manager.go:414] "containerMap: removing container" podUID="adb403b1-bc1d-4f49-88e1-fbb1baceae02" containerName="maas-api"
Oct 30 20:12:06.763323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763126    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb403b1-bc1d-4f49-88e1-fbb1baceae02" containerName="maas-api"
Oct 30 20:12:06.763323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763135    4823 cpu_manager.go:414] "containerMap: removing container" podUID="3c6e7d8f-a41b-4609-a6b4-e0642890b4b4" containerName="revoke-keys"
Oct 30 20:12:06.763323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763140    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6e7d8f-a41b-4609-a6b4-e0642890b4b4" containerName="revoke-keys"
Oct 30 20:12:06.763323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763150    4823 cpu_manager.go:414] "containerMap: removing container" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main"
Oct 30 20:12:06.763323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763154    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main"
Oct 30 20:12:06.763323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763253    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c5b746-9543-43b1-b132-ebe7ba633d9d" containerName="revoke-keys"
Oct 30 20:12:06.763323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763264    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" containerName="istio-proxy"
Oct 30 20:12:06.763323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763271    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="db51eeaa-23d4-4ade-9e6e-6c59fae623c0" containerName="revoke-keys"
Oct 30 20:12:06.763323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763280    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb403b1-bc1d-4f49-88e1-fbb1baceae02" containerName="maas-api"
Oct 30 20:12:06.763323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763286    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6e7d8f-a41b-4609-a6b4-e0642890b4b4" containerName="revoke-keys"
Oct 30 20:12:06.763323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.763292    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd230496-5fbe-4500-965c-16e48f099552" containerName="main"
Oct 30 20:12:06.768741 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.768717    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"
Oct 30 20:12:06.774368 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.774221    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:12:06.774368 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.774334    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:12:06.774530 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.774463    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:12:06.775145 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.774917    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:12:06.776812 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.776785    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"]
Oct 30 20:12:06.812691 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.812655    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c8faa2b-89f0-4083-b46a-02cdbac45649-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw\" (UID: \"1c8faa2b-89f0-4083-b46a-02cdbac45649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"
Oct 30 20:12:06.812901 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.812828    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzpqp\" (UniqueName: \"kubernetes.io/projected/1c8faa2b-89f0-4083-b46a-02cdbac45649-kube-api-access-zzpqp\") pod \"maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw\" (UID: \"1c8faa2b-89f0-4083-b46a-02cdbac45649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"
Oct 30 20:12:06.813074 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.813052    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c8faa2b-89f0-4083-b46a-02cdbac45649-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw\" (UID: \"1c8faa2b-89f0-4083-b46a-02cdbac45649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"
Oct 30 20:12:06.915288 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.915248    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c8faa2b-89f0-4083-b46a-02cdbac45649-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw\" (UID: \"1c8faa2b-89f0-4083-b46a-02cdbac45649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"
Oct 30 20:12:06.915470 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.915361    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzpqp\" (UniqueName: \"kubernetes.io/projected/1c8faa2b-89f0-4083-b46a-02cdbac45649-kube-api-access-zzpqp\") pod \"maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw\" (UID: \"1c8faa2b-89f0-4083-b46a-02cdbac45649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"
Oct 30 20:12:06.915524 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.915504    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c8faa2b-89f0-4083-b46a-02cdbac45649-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw\" (UID: \"1c8faa2b-89f0-4083-b46a-02cdbac45649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"
Oct 30 20:12:06.916191 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.916165    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c8faa2b-89f0-4083-b46a-02cdbac45649-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw\" (UID: \"1c8faa2b-89f0-4083-b46a-02cdbac45649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"
Oct 30 20:12:06.917663 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.917640    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c8faa2b-89f0-4083-b46a-02cdbac45649-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw\" (UID: \"1c8faa2b-89f0-4083-b46a-02cdbac45649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"
Oct 30 20:12:06.947359 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:06.947337    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzpqp\" (UniqueName: \"kubernetes.io/projected/1c8faa2b-89f0-4083-b46a-02cdbac45649-kube-api-access-zzpqp\") pod \"maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw\" (UID: \"1c8faa2b-89f0-4083-b46a-02cdbac45649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"
Oct 30 20:12:07.086398 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:07.086322    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"
Oct 30 20:12:07.374274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:07.374244    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"]
Oct 30 20:12:07.374610 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:12:07.374548    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8faa2b_89f0_4083_b46a_02cdbac45649.slice/crio-fc5f5308845809b12771623e99368eafc9b131a449b1262f760a6f41cb997d0e WatchSource:0}: Error finding container fc5f5308845809b12771623e99368eafc9b131a449b1262f760a6f41cb997d0e: Status 404 returned error can't find the container with id fc5f5308845809b12771623e99368eafc9b131a449b1262f760a6f41cb997d0e
Oct 30 20:12:08.117307 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:08.117268    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw" event={"ID":"1c8faa2b-89f0-4083-b46a-02cdbac45649","Type":"ContainerStarted","Data":"dabef7919df01b48ba17e2eb6991a22a90017fcce8fdcd28f0f63fa7eb6b8bda"}
Oct 30 20:12:08.117477 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:08.117334    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw" event={"ID":"1c8faa2b-89f0-4083-b46a-02cdbac45649","Type":"ContainerStarted","Data":"fc5f5308845809b12771623e99368eafc9b131a449b1262f760a6f41cb997d0e"}
Oct 30 20:12:08.144591 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:08.144534    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw" podStartSLOduration=2.144519838 podStartE2EDuration="2.144519838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:08.139616402 +0000 UTC m=+2528.032230725" watchObservedRunningTime="2026-10-30 20:12:08.144519838 +0000 UTC m=+2528.037134221"
Oct 30 20:12:08.352396 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:08.352363    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm" podUID="fd0d4ac0-27c1-4735-b333-63b4e75afe70" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:12:09.126573 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:09.126533    4823 generic.go:356] "Generic (PLEG): container finished" podID="1c8faa2b-89f0-4083-b46a-02cdbac45649" containerID="dabef7919df01b48ba17e2eb6991a22a90017fcce8fdcd28f0f63fa7eb6b8bda" exitCode=0
Oct 30 20:12:09.126980 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:09.126594    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw" event={"ID":"1c8faa2b-89f0-4083-b46a-02cdbac45649","Type":"ContainerDied","Data":"dabef7919df01b48ba17e2eb6991a22a90017fcce8fdcd28f0f63fa7eb6b8bda"}
Oct 30 20:12:09.261089 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:09.261055    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"]
Oct 30 20:12:09.261424 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:09.261373    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" containerID="cri-o://b68f42c20d5aa0671e1cde35cab2bc0268b4f972805f2c3823a7c9a51c637f3e" gracePeriod=30
Oct 30 20:12:10.301296 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:10.301273    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"
Oct 30 20:12:10.418566 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:10.418482    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c8faa2b-89f0-4083-b46a-02cdbac45649-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c8faa2b-89f0-4083-b46a-02cdbac45649-maas-api-openshift-ca\") pod \"1c8faa2b-89f0-4083-b46a-02cdbac45649\" (UID: \"1c8faa2b-89f0-4083-b46a-02cdbac45649\") "
Oct 30 20:12:10.418566 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:10.418531    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8faa2b-89f0-4083-b46a-02cdbac45649-kube-api-access-zzpqp\" (UniqueName: \"kubernetes.io/projected/1c8faa2b-89f0-4083-b46a-02cdbac45649-kube-api-access-zzpqp\") pod \"1c8faa2b-89f0-4083-b46a-02cdbac45649\" (UID: \"1c8faa2b-89f0-4083-b46a-02cdbac45649\") "
Oct 30 20:12:10.418773 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:10.418585    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c8faa2b-89f0-4083-b46a-02cdbac45649-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c8faa2b-89f0-4083-b46a-02cdbac45649-maas-api-xks-ca\") pod \"1c8faa2b-89f0-4083-b46a-02cdbac45649\" (UID: \"1c8faa2b-89f0-4083-b46a-02cdbac45649\") "
Oct 30 20:12:10.418921 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:10.418853    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8faa2b-89f0-4083-b46a-02cdbac45649-maas-api-openshift-ca" pod "1c8faa2b-89f0-4083-b46a-02cdbac45649" (UID: "1c8faa2b-89f0-4083-b46a-02cdbac45649"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:10.419393 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:10.419368    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c8faa2b-89f0-4083-b46a-02cdbac45649-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:10.420807 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:10.420778    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c8faa2b-89f0-4083-b46a-02cdbac45649-maas-api-xks-ca" pod "1c8faa2b-89f0-4083-b46a-02cdbac45649" (UID: "1c8faa2b-89f0-4083-b46a-02cdbac45649"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:10.420946 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:10.420810    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8faa2b-89f0-4083-b46a-02cdbac45649-kube-api-access-zzpqp" pod "1c8faa2b-89f0-4083-b46a-02cdbac45649" (UID: "1c8faa2b-89f0-4083-b46a-02cdbac45649"). InnerVolumeSpecName "kube-api-access-zzpqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:10.521456 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:10.521417    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzpqp\" (UniqueName: \"kubernetes.io/projected/1c8faa2b-89f0-4083-b46a-02cdbac45649-kube-api-access-zzpqp\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:10.521456 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:10.521449    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c8faa2b-89f0-4083-b46a-02cdbac45649-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:11.146309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.146276    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"
Oct 30 20:12:11.146521 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.146306    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw" event={"ID":"1c8faa2b-89f0-4083-b46a-02cdbac45649","Type":"ContainerDied","Data":"fc5f5308845809b12771623e99368eafc9b131a449b1262f760a6f41cb997d0e"}
Oct 30 20:12:11.146521 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.146397    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc5f5308845809b12771623e99368eafc9b131a449b1262f760a6f41cb997d0e"
Oct 30 20:12:11.245762 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.245728    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"]
Oct 30 20:12:11.249740 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.249712    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a1d885-j9g4p"]
Oct 30 20:12:11.588913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.588830    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"]
Oct 30 20:12:11.589540 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.589523    4823 cpu_manager.go:414] "containerMap: removing container" podUID="1c8faa2b-89f0-4083-b46a-02cdbac45649" containerName="revoke-keys"
Oct 30 20:12:11.589540 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.589539    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8faa2b-89f0-4083-b46a-02cdbac45649" containerName="revoke-keys"
Oct 30 20:12:11.589744 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.589650    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f06d686-b1bf-48e3-b9ee-cdc0d232ecd3" containerName="istio-proxy"
Oct 30 20:12:11.589744 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.589662    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8faa2b-89f0-4083-b46a-02cdbac45649" containerName="revoke-keys"
Oct 30 20:12:11.593220 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.593204    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"
Oct 30 20:12:11.597635 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.597607    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:12:11.601797 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.599838    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:12:11.606703 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.606669    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"]
Oct 30 20:12:11.662084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.662049    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52af467d-0e62-456d-b43c-016003f6e0b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8\" (UID: \"52af467d-0e62-456d-b43c-016003f6e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"
Oct 30 20:12:11.662294 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.662176    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj9z7\" (UniqueName: \"kubernetes.io/projected/52af467d-0e62-456d-b43c-016003f6e0b4-kube-api-access-xj9z7\") pod \"maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8\" (UID: \"52af467d-0e62-456d-b43c-016003f6e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"
Oct 30 20:12:11.662376 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.662294    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52af467d-0e62-456d-b43c-016003f6e0b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8\" (UID: \"52af467d-0e62-456d-b43c-016003f6e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"
Oct 30 20:12:11.663746 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.663720    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:12:11.663996 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.663979    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:12:11.763585 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.763556    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52af467d-0e62-456d-b43c-016003f6e0b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8\" (UID: \"52af467d-0e62-456d-b43c-016003f6e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"
Oct 30 20:12:11.763745 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.763646    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj9z7\" (UniqueName: \"kubernetes.io/projected/52af467d-0e62-456d-b43c-016003f6e0b4-kube-api-access-xj9z7\") pod \"maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8\" (UID: \"52af467d-0e62-456d-b43c-016003f6e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"
Oct 30 20:12:11.763745 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.763711    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52af467d-0e62-456d-b43c-016003f6e0b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8\" (UID: \"52af467d-0e62-456d-b43c-016003f6e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"
Oct 30 20:12:11.764841 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.764817    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52af467d-0e62-456d-b43c-016003f6e0b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8\" (UID: \"52af467d-0e62-456d-b43c-016003f6e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"
Oct 30 20:12:11.766493 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.766469    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52af467d-0e62-456d-b43c-016003f6e0b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8\" (UID: \"52af467d-0e62-456d-b43c-016003f6e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"
Oct 30 20:12:11.777065 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.777036    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj9z7\" (UniqueName: \"kubernetes.io/projected/52af467d-0e62-456d-b43c-016003f6e0b4-kube-api-access-xj9z7\") pod \"maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8\" (UID: \"52af467d-0e62-456d-b43c-016003f6e0b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"
Oct 30 20:12:11.912390 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:11.912359    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"
Oct 30 20:12:12.358949 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:12.358902    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86101de1-ddcb-4d84-b487-7221a42ee4db" path="/var/lib/kubelet/pods/86101de1-ddcb-4d84-b487-7221a42ee4db/volumes"
Oct 30 20:12:12.360017 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:12.359996    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"]
Oct 30 20:12:13.212674 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:13.212635    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8" event={"ID":"52af467d-0e62-456d-b43c-016003f6e0b4","Type":"ContainerStarted","Data":"60c94502bd743d2c3f091b0e7ccb7678c4ecbf490d8c0ea627a5f295bb334bbe"}
Oct 30 20:12:13.213215 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:13.212688    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8" event={"ID":"52af467d-0e62-456d-b43c-016003f6e0b4","Type":"ContainerStarted","Data":"2065547e58e4e4607194d970d613b918619285884195bdbc13988efb048569c6"}
Oct 30 20:12:13.240217 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:13.240166    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8" podStartSLOduration=2.240151876 podStartE2EDuration="2.240151876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:13.237677209 +0000 UTC m=+2533.130291534" watchObservedRunningTime="2026-10-30 20:12:13.240151876 +0000 UTC m=+2533.132766197"
Oct 30 20:12:13.577172 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:13.577090    4823 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:12:13.577172 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:13.577145    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:12:14.237239 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.237203    4823 generic.go:356] "Generic (PLEG): container finished" podID="52af467d-0e62-456d-b43c-016003f6e0b4" containerID="60c94502bd743d2c3f091b0e7ccb7678c4ecbf490d8c0ea627a5f295bb334bbe" exitCode=0
Oct 30 20:12:14.237648 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.237283    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8" event={"ID":"52af467d-0e62-456d-b43c-016003f6e0b4","Type":"ContainerDied","Data":"60c94502bd743d2c3f091b0e7ccb7678c4ecbf490d8c0ea627a5f295bb334bbe"}
Oct 30 20:12:14.468832 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.468795    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"]
Oct 30 20:12:14.469074 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.469049    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95" podUID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" containerName="istio-proxy" containerID="cri-o://1e88f27d7c5878928f0ae468e4a054544adf45912e600799202b2d33c8248651" gracePeriod=30
Oct 30 20:12:14.685886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.685067    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 20:12:14.763883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.758443    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6adff78e-1673-4bee-98b9-71771700b23f-kube-api-access-b64h6\" (UniqueName: \"kubernetes.io/projected/6adff78e-1673-4bee-98b9-71771700b23f-kube-api-access-b64h6\") pod \"6adff78e-1673-4bee-98b9-71771700b23f\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") "
Oct 30 20:12:14.763883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.758515    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-workload-certs\") pod \"6adff78e-1673-4bee-98b9-71771700b23f\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") "
Oct 30 20:12:14.763883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.758551    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-workload-socket\") pod \"6adff78e-1673-4bee-98b9-71771700b23f\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") "
Oct 30 20:12:14.763883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.758683    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-istio-data\") pod \"6adff78e-1673-4bee-98b9-71771700b23f\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") "
Oct 30 20:12:14.763883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.758739    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6adff78e-1673-4bee-98b9-71771700b23f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6adff78e-1673-4bee-98b9-71771700b23f-istio-podinfo\") pod \"6adff78e-1673-4bee-98b9-71771700b23f\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") "
Oct 30 20:12:14.763883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.758782    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6adff78e-1673-4bee-98b9-71771700b23f-istio-token\" (UniqueName: \"kubernetes.io/projected/6adff78e-1673-4bee-98b9-71771700b23f-istio-token\") pod \"6adff78e-1673-4bee-98b9-71771700b23f\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") "
Oct 30 20:12:14.763883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.758818    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6adff78e-1673-4bee-98b9-71771700b23f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6adff78e-1673-4bee-98b9-71771700b23f-istiod-ca-cert\") pod \"6adff78e-1673-4bee-98b9-71771700b23f\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") "
Oct 30 20:12:14.763883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.758851    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-istio-envoy\") pod \"6adff78e-1673-4bee-98b9-71771700b23f\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") "
Oct 30 20:12:14.763883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.758904    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-credential-socket\") pod \"6adff78e-1673-4bee-98b9-71771700b23f\" (UID: \"6adff78e-1673-4bee-98b9-71771700b23f\") "
Oct 30 20:12:14.766898 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.766852    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6adff78e-1673-4bee-98b9-71771700b23f-istio-podinfo" pod "6adff78e-1673-4bee-98b9-71771700b23f" (UID: "6adff78e-1673-4bee-98b9-71771700b23f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:12:14.768635 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.768043    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6adff78e-1673-4bee-98b9-71771700b23f-istio-token" pod "6adff78e-1673-4bee-98b9-71771700b23f" (UID: "6adff78e-1673-4bee-98b9-71771700b23f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:14.769295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.768877    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-workload-certs" pod "6adff78e-1673-4bee-98b9-71771700b23f" (UID: "6adff78e-1673-4bee-98b9-71771700b23f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:14.771020 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.770996    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-workload-socket" pod "6adff78e-1673-4bee-98b9-71771700b23f" (UID: "6adff78e-1673-4bee-98b9-71771700b23f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:14.771615 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.771593    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6adff78e-1673-4bee-98b9-71771700b23f-istiod-ca-cert" pod "6adff78e-1673-4bee-98b9-71771700b23f" (UID: "6adff78e-1673-4bee-98b9-71771700b23f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:14.772405 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.772367    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-istio-data" pod "6adff78e-1673-4bee-98b9-71771700b23f" (UID: "6adff78e-1673-4bee-98b9-71771700b23f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:14.790988 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.790954    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-credential-socket" pod "6adff78e-1673-4bee-98b9-71771700b23f" (UID: "6adff78e-1673-4bee-98b9-71771700b23f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:14.805355 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.805319    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6adff78e-1673-4bee-98b9-71771700b23f-kube-api-access-b64h6" pod "6adff78e-1673-4bee-98b9-71771700b23f" (UID: "6adff78e-1673-4bee-98b9-71771700b23f"). InnerVolumeSpecName "kube-api-access-b64h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:14.806801 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.806774    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-istio-envoy" pod "6adff78e-1673-4bee-98b9-71771700b23f" (UID: "6adff78e-1673-4bee-98b9-71771700b23f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:14.861295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.861073    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b64h6\" (UniqueName: \"kubernetes.io/projected/6adff78e-1673-4bee-98b9-71771700b23f-kube-api-access-b64h6\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.861295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.861107    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.861295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.861120    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.861295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.861129    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.861295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.861139    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6adff78e-1673-4bee-98b9-71771700b23f-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.861295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.861148    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6adff78e-1673-4bee-98b9-71771700b23f-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.861295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.861157    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6adff78e-1673-4bee-98b9-71771700b23f-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.861295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.861165    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:14.861295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:14.861175    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6adff78e-1673-4bee-98b9-71771700b23f-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:15.191819 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.191743    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"]
Oct 30 20:12:15.192604 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.192576    4823 cpu_manager.go:414] "containerMap: removing container" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy"
Oct 30 20:12:15.192604 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.192599    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy"
Oct 30 20:12:15.192771 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.192725    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="6adff78e-1673-4bee-98b9-71771700b23f" containerName="istio-proxy"
Oct 30 20:12:15.196564 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.196543    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"
Oct 30 20:12:15.216465 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.216434    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"]
Oct 30 20:12:15.248266 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.248229    4823 generic.go:356] "Generic (PLEG): container finished" podID="6adff78e-1673-4bee-98b9-71771700b23f" containerID="b68f42c20d5aa0671e1cde35cab2bc0268b4f972805f2c3823a7c9a51c637f3e" exitCode=0
Oct 30 20:12:15.248708 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.248311    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" event={"ID":"6adff78e-1673-4bee-98b9-71771700b23f","Type":"ContainerDied","Data":"b68f42c20d5aa0671e1cde35cab2bc0268b4f972805f2c3823a7c9a51c637f3e"}
Oct 30 20:12:15.248708 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.248342    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"
Oct 30 20:12:15.248708 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.248372    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks" event={"ID":"6adff78e-1673-4bee-98b9-71771700b23f","Type":"ContainerDied","Data":"70d838ad0c7217853e6bf3fccc7d469029dc7cc2b15edcc30fc28949847d66c2"}
Oct 30 20:12:15.248708 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.248397    4823 scope.go:122] "RemoveContainer" containerID="b68f42c20d5aa0671e1cde35cab2bc0268b4f972805f2c3823a7c9a51c637f3e"
Oct 30 20:12:15.261530 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.261511    4823 scope.go:122] "RemoveContainer" containerID="b68f42c20d5aa0671e1cde35cab2bc0268b4f972805f2c3823a7c9a51c637f3e"
Oct 30 20:12:15.261926 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:12:15.261851    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68f42c20d5aa0671e1cde35cab2bc0268b4f972805f2c3823a7c9a51c637f3e\": container with ID starting with b68f42c20d5aa0671e1cde35cab2bc0268b4f972805f2c3823a7c9a51c637f3e not found: ID does not exist" containerID="b68f42c20d5aa0671e1cde35cab2bc0268b4f972805f2c3823a7c9a51c637f3e"
Oct 30 20:12:15.262019 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.261940    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68f42c20d5aa0671e1cde35cab2bc0268b4f972805f2c3823a7c9a51c637f3e"} err="failed to get container status \"b68f42c20d5aa0671e1cde35cab2bc0268b4f972805f2c3823a7c9a51c637f3e\": rpc error: code = NotFound desc = could not find container \"b68f42c20d5aa0671e1cde35cab2bc0268b4f972805f2c3823a7c9a51c637f3e\": container with ID starting with b68f42c20d5aa0671e1cde35cab2bc0268b4f972805f2c3823a7c9a51c637f3e not found: ID does not exist"
Oct 30 20:12:15.266187 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.266159    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712538e4-19f3-4e54-94ab-d6b435581839-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq\" (UID: \"712538e4-19f3-4e54-94ab-d6b435581839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"
Oct 30 20:12:15.266308 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.266245    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72gh7\" (UniqueName: \"kubernetes.io/projected/712538e4-19f3-4e54-94ab-d6b435581839-kube-api-access-72gh7\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq\" (UID: \"712538e4-19f3-4e54-94ab-d6b435581839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"
Oct 30 20:12:15.266308 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.266266    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712538e4-19f3-4e54-94ab-d6b435581839-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq\" (UID: \"712538e4-19f3-4e54-94ab-d6b435581839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"
Oct 30 20:12:15.283027 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.282970    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"]
Oct 30 20:12:15.287263 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.287238    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-hx9ks"]
Oct 30 20:12:15.368339 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.368286    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72gh7\" (UniqueName: \"kubernetes.io/projected/712538e4-19f3-4e54-94ab-d6b435581839-kube-api-access-72gh7\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq\" (UID: \"712538e4-19f3-4e54-94ab-d6b435581839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"
Oct 30 20:12:15.394989 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.394949    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712538e4-19f3-4e54-94ab-d6b435581839-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq\" (UID: \"712538e4-19f3-4e54-94ab-d6b435581839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"
Oct 30 20:12:15.411066 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.396144    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712538e4-19f3-4e54-94ab-d6b435581839-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq\" (UID: \"712538e4-19f3-4e54-94ab-d6b435581839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"
Oct 30 20:12:15.412407 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.412380    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712538e4-19f3-4e54-94ab-d6b435581839-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq\" (UID: \"712538e4-19f3-4e54-94ab-d6b435581839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"
Oct 30 20:12:15.419279 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.413478    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72gh7\" (UniqueName: \"kubernetes.io/projected/712538e4-19f3-4e54-94ab-d6b435581839-kube-api-access-72gh7\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq\" (UID: \"712538e4-19f3-4e54-94ab-d6b435581839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"
Oct 30 20:12:15.424226 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.422405    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712538e4-19f3-4e54-94ab-d6b435581839-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq\" (UID: \"712538e4-19f3-4e54-94ab-d6b435581839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"
Oct 30 20:12:15.475209 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.475185    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"
Oct 30 20:12:15.511724 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.511690    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"
Oct 30 20:12:15.615570 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.615540    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52af467d-0e62-456d-b43c-016003f6e0b4-kube-api-access-xj9z7\" (UniqueName: \"kubernetes.io/projected/52af467d-0e62-456d-b43c-016003f6e0b4-kube-api-access-xj9z7\") pod \"52af467d-0e62-456d-b43c-016003f6e0b4\" (UID: \"52af467d-0e62-456d-b43c-016003f6e0b4\") "
Oct 30 20:12:15.615746 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.615669    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52af467d-0e62-456d-b43c-016003f6e0b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52af467d-0e62-456d-b43c-016003f6e0b4-maas-api-xks-ca\") pod \"52af467d-0e62-456d-b43c-016003f6e0b4\" (UID: \"52af467d-0e62-456d-b43c-016003f6e0b4\") "
Oct 30 20:12:15.615746 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.615695    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52af467d-0e62-456d-b43c-016003f6e0b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52af467d-0e62-456d-b43c-016003f6e0b4-maas-api-openshift-ca\") pod \"52af467d-0e62-456d-b43c-016003f6e0b4\" (UID: \"52af467d-0e62-456d-b43c-016003f6e0b4\") "
Oct 30 20:12:15.616986 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.616960    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52af467d-0e62-456d-b43c-016003f6e0b4-maas-api-openshift-ca" pod "52af467d-0e62-456d-b43c-016003f6e0b4" (UID: "52af467d-0e62-456d-b43c-016003f6e0b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:15.618840 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.618811    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52af467d-0e62-456d-b43c-016003f6e0b4-maas-api-xks-ca" pod "52af467d-0e62-456d-b43c-016003f6e0b4" (UID: "52af467d-0e62-456d-b43c-016003f6e0b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:15.618975 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.618917    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52af467d-0e62-456d-b43c-016003f6e0b4-kube-api-access-xj9z7" pod "52af467d-0e62-456d-b43c-016003f6e0b4" (UID: "52af467d-0e62-456d-b43c-016003f6e0b4"). InnerVolumeSpecName "kube-api-access-xj9z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:15.718274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.718195    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj9z7\" (UniqueName: \"kubernetes.io/projected/52af467d-0e62-456d-b43c-016003f6e0b4-kube-api-access-xj9z7\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:15.718274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.718225    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52af467d-0e62-456d-b43c-016003f6e0b4-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:15.718274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.718235    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52af467d-0e62-456d-b43c-016003f6e0b4-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:15.808588 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:15.808557    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"]
Oct 30 20:12:15.809442 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:12:15.809413    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712538e4_19f3_4e54_94ab_d6b435581839.slice/crio-a562f9271816cfbd3839cd2c6d958505dbfd3df9030f4a7809f1da03ad70a2dc WatchSource:0}: Error finding container a562f9271816cfbd3839cd2c6d958505dbfd3df9030f4a7809f1da03ad70a2dc: Status 404 returned error can't find the container with id a562f9271816cfbd3839cd2c6d958505dbfd3df9030f4a7809f1da03ad70a2dc
Oct 30 20:12:16.259782 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:16.259740    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq" event={"ID":"712538e4-19f3-4e54-94ab-d6b435581839","Type":"ContainerStarted","Data":"ad934b8ce228338c98bc334cec9dc4f4eb0505b59e760936d997fe6039191a22"}
Oct 30 20:12:16.260253 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:16.259807    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq" event={"ID":"712538e4-19f3-4e54-94ab-d6b435581839","Type":"ContainerStarted","Data":"a562f9271816cfbd3839cd2c6d958505dbfd3df9030f4a7809f1da03ad70a2dc"}
Oct 30 20:12:16.261390 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:16.261366    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"
Oct 30 20:12:16.261484 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:16.261393    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8" event={"ID":"52af467d-0e62-456d-b43c-016003f6e0b4","Type":"ContainerDied","Data":"2065547e58e4e4607194d970d613b918619285884195bdbc13988efb048569c6"}
Oct 30 20:12:16.261484 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:16.261435    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2065547e58e4e4607194d970d613b918619285884195bdbc13988efb048569c6"
Oct 30 20:12:16.279770 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:16.279744    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6adff78e-1673-4bee-98b9-71771700b23f" path="/var/lib/kubelet/pods/6adff78e-1673-4bee-98b9-71771700b23f/volumes"
Oct 30 20:12:16.290729 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:16.290666    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq" podStartSLOduration=1.290653173 podStartE2EDuration="1.290653173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:16.28518245 +0000 UTC m=+2536.177796774" watchObservedRunningTime="2026-10-30 20:12:16.290653173 +0000 UTC m=+2536.183267496"
Oct 30 20:12:16.385895 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:16.385829    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"]
Oct 30 20:12:16.422173 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:16.422136    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"]
Oct 30 20:12:17.419885 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.411218    4823 generic.go:356] "Generic (PLEG): container finished" podID="712538e4-19f3-4e54-94ab-d6b435581839" containerID="ad934b8ce228338c98bc334cec9dc4f4eb0505b59e760936d997fe6039191a22" exitCode=0
Oct 30 20:12:17.419885 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.411639    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq" event={"ID":"712538e4-19f3-4e54-94ab-d6b435581839","Type":"ContainerDied","Data":"ad934b8ce228338c98bc334cec9dc4f4eb0505b59e760936d997fe6039191a22"}
Oct 30 20:12:17.453143 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.428930    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"]
Oct 30 20:12:17.453143 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.430714    4823 cpu_manager.go:414] "containerMap: removing container" podUID="52af467d-0e62-456d-b43c-016003f6e0b4" containerName="revoke-keys"
Oct 30 20:12:17.453143 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.430737    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52af467d-0e62-456d-b43c-016003f6e0b4" containerName="revoke-keys"
Oct 30 20:12:17.453143 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.431047    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="52af467d-0e62-456d-b43c-016003f6e0b4" containerName="revoke-keys"
Oct 30 20:12:17.478278 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.472727    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"
Oct 30 20:12:17.497072 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.497030    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"]
Oct 30 20:12:17.578265 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.578210    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/669b522e-bca0-45a3-a6c5-b79a3460414a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl\" (UID: \"669b522e-bca0-45a3-a6c5-b79a3460414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"
Oct 30 20:12:17.578452 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.578328    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/669b522e-bca0-45a3-a6c5-b79a3460414a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl\" (UID: \"669b522e-bca0-45a3-a6c5-b79a3460414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"
Oct 30 20:12:17.578565 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.578541    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ctgl\" (UniqueName: \"kubernetes.io/projected/669b522e-bca0-45a3-a6c5-b79a3460414a-kube-api-access-8ctgl\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl\" (UID: \"669b522e-bca0-45a3-a6c5-b79a3460414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"
Oct 30 20:12:17.679778 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.679684    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ctgl\" (UniqueName: \"kubernetes.io/projected/669b522e-bca0-45a3-a6c5-b79a3460414a-kube-api-access-8ctgl\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl\" (UID: \"669b522e-bca0-45a3-a6c5-b79a3460414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"
Oct 30 20:12:17.679986 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.679922    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/669b522e-bca0-45a3-a6c5-b79a3460414a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl\" (UID: \"669b522e-bca0-45a3-a6c5-b79a3460414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"
Oct 30 20:12:17.679986 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.679970    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/669b522e-bca0-45a3-a6c5-b79a3460414a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl\" (UID: \"669b522e-bca0-45a3-a6c5-b79a3460414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"
Oct 30 20:12:17.680726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.680701    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/669b522e-bca0-45a3-a6c5-b79a3460414a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl\" (UID: \"669b522e-bca0-45a3-a6c5-b79a3460414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"
Oct 30 20:12:17.683599 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.683575    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/669b522e-bca0-45a3-a6c5-b79a3460414a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl\" (UID: \"669b522e-bca0-45a3-a6c5-b79a3460414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"
Oct 30 20:12:17.692738 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.692704    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ctgl\" (UniqueName: \"kubernetes.io/projected/669b522e-bca0-45a3-a6c5-b79a3460414a-kube-api-access-8ctgl\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl\" (UID: \"669b522e-bca0-45a3-a6c5-b79a3460414a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"
Oct 30 20:12:17.808161 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:17.808122    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"
Oct 30 20:12:18.115553 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:18.115522    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"]
Oct 30 20:12:18.121787 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:18.121750    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-77b4df-7thhw"]
Oct 30 20:12:18.280098 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:18.280002    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c8faa2b-89f0-4083-b46a-02cdbac45649" path="/var/lib/kubelet/pods/1c8faa2b-89f0-4083-b46a-02cdbac45649/volumes"
Oct 30 20:12:18.284929 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:18.284904    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"]
Oct 30 20:12:18.285163 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:12:18.285141    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669b522e_bca0_45a3_a6c5_b79a3460414a.slice/crio-548514322c1ee322a242ef7ca7529a931a489fe07b9fc9f5856f0fe01c6c2115 WatchSource:0}: Error finding container 548514322c1ee322a242ef7ca7529a931a489fe07b9fc9f5856f0fe01c6c2115: Status 404 returned error can't find the container with id 548514322c1ee322a242ef7ca7529a931a489fe07b9fc9f5856f0fe01c6c2115
Oct 30 20:12:18.351802 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:18.351769    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm" podUID="fd0d4ac0-27c1-4735-b333-63b4e75afe70" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:12:18.420527 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:18.420496    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl" event={"ID":"669b522e-bca0-45a3-a6c5-b79a3460414a","Type":"ContainerStarted","Data":"548514322c1ee322a242ef7ca7529a931a489fe07b9fc9f5856f0fe01c6c2115"}
Oct 30 20:12:18.561341 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:18.561317    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"
Oct 30 20:12:18.656379 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:18.656349    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/712538e4-19f3-4e54-94ab-d6b435581839-kube-api-access-72gh7\" (UniqueName: \"kubernetes.io/projected/712538e4-19f3-4e54-94ab-d6b435581839-kube-api-access-72gh7\") pod \"712538e4-19f3-4e54-94ab-d6b435581839\" (UID: \"712538e4-19f3-4e54-94ab-d6b435581839\") "
Oct 30 20:12:18.656564 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:18.656462    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/712538e4-19f3-4e54-94ab-d6b435581839-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712538e4-19f3-4e54-94ab-d6b435581839-maas-api-openshift-ca\") pod \"712538e4-19f3-4e54-94ab-d6b435581839\" (UID: \"712538e4-19f3-4e54-94ab-d6b435581839\") "
Oct 30 20:12:18.656564 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:18.656552    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/712538e4-19f3-4e54-94ab-d6b435581839-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712538e4-19f3-4e54-94ab-d6b435581839-maas-api-xks-ca\") pod \"712538e4-19f3-4e54-94ab-d6b435581839\" (UID: \"712538e4-19f3-4e54-94ab-d6b435581839\") "
Oct 30 20:12:18.656929 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:18.656850    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712538e4-19f3-4e54-94ab-d6b435581839-maas-api-openshift-ca" pod "712538e4-19f3-4e54-94ab-d6b435581839" (UID: "712538e4-19f3-4e54-94ab-d6b435581839"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:18.657310 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:18.657290    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712538e4-19f3-4e54-94ab-d6b435581839-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:18.658613 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:18.658591    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712538e4-19f3-4e54-94ab-d6b435581839-kube-api-access-72gh7" pod "712538e4-19f3-4e54-94ab-d6b435581839" (UID: "712538e4-19f3-4e54-94ab-d6b435581839"). InnerVolumeSpecName "kube-api-access-72gh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:18.658687 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:18.658628    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712538e4-19f3-4e54-94ab-d6b435581839-maas-api-xks-ca" pod "712538e4-19f3-4e54-94ab-d6b435581839" (UID: "712538e4-19f3-4e54-94ab-d6b435581839"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:18.758901 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:18.758833    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72gh7\" (UniqueName: \"kubernetes.io/projected/712538e4-19f3-4e54-94ab-d6b435581839-kube-api-access-72gh7\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:18.758901 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:18.758887    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712538e4-19f3-4e54-94ab-d6b435581839-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:19.267256 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.267228    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"
Oct 30 20:12:19.370922 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.367659    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd0d4ac0-27c1-4735-b333-63b4e75afe70-plugins-config-volume" pod "fd0d4ac0-27c1-4735-b333-63b4e75afe70" (UID: "fd0d4ac0-27c1-4735-b333-63b4e75afe70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:19.370922 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.367737    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd0d4ac0-27c1-4735-b333-63b4e75afe70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0d4ac0-27c1-4735-b333-63b4e75afe70-plugins-config-volume\") pod \"fd0d4ac0-27c1-4735-b333-63b4e75afe70\" (UID: \"fd0d4ac0-27c1-4735-b333-63b4e75afe70\") "
Oct 30 20:12:19.370922 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.368096    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd0d4ac0-27c1-4735-b333-63b4e75afe70-kube-api-access-6dl6n\" (UniqueName: \"kubernetes.io/projected/fd0d4ac0-27c1-4735-b333-63b4e75afe70-kube-api-access-6dl6n\") pod \"fd0d4ac0-27c1-4735-b333-63b4e75afe70\" (UID: \"fd0d4ac0-27c1-4735-b333-63b4e75afe70\") "
Oct 30 20:12:19.371156 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.371057    4823 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd0d4ac0-27c1-4735-b333-63b4e75afe70-plugins-config-volume\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:19.380427 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.380391    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0d4ac0-27c1-4735-b333-63b4e75afe70-kube-api-access-6dl6n" pod "fd0d4ac0-27c1-4735-b333-63b4e75afe70" (UID: "fd0d4ac0-27c1-4735-b333-63b4e75afe70"). InnerVolumeSpecName "kube-api-access-6dl6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:19.430935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.430832    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq" event={"ID":"712538e4-19f3-4e54-94ab-d6b435581839","Type":"ContainerDied","Data":"a562f9271816cfbd3839cd2c6d958505dbfd3df9030f4a7809f1da03ad70a2dc"}
Oct 30 20:12:19.430935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.430841    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"
Oct 30 20:12:19.430935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.430897    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a562f9271816cfbd3839cd2c6d958505dbfd3df9030f4a7809f1da03ad70a2dc"
Oct 30 20:12:19.432556 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.432530    4823 generic.go:356] "Generic (PLEG): container finished" podID="fd0d4ac0-27c1-4735-b333-63b4e75afe70" containerID="635dad2452b6b4b06f14eb51e27df8beae0404ddccb230204de15dff6d17c1b4" exitCode=0
Oct 30 20:12:19.432699 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.432566    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"
Oct 30 20:12:19.432699 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.432605    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm" event={"ID":"fd0d4ac0-27c1-4735-b333-63b4e75afe70","Type":"ContainerDied","Data":"635dad2452b6b4b06f14eb51e27df8beae0404ddccb230204de15dff6d17c1b4"}
Oct 30 20:12:19.432699 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.432650    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm" event={"ID":"fd0d4ac0-27c1-4735-b333-63b4e75afe70","Type":"ContainerDied","Data":"f08ada36b720a6941ccb3e8594a40f3cc5dfe8e2df5876cbf3657ce2a5ca8b1f"}
Oct 30 20:12:19.432699 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.432673    4823 scope.go:122] "RemoveContainer" containerID="635dad2452b6b4b06f14eb51e27df8beae0404ddccb230204de15dff6d17c1b4"
Oct 30 20:12:19.439098 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.439068    4823 generic.go:356] "Generic (PLEG): container finished" podID="669b522e-bca0-45a3-a6c5-b79a3460414a" containerID="148afa2990ea874183769d17a1a27b40b5e40730b77f0929847ea2e6772efdd1" exitCode=0
Oct 30 20:12:19.439233 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.439119    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl" event={"ID":"669b522e-bca0-45a3-a6c5-b79a3460414a","Type":"ContainerDied","Data":"148afa2990ea874183769d17a1a27b40b5e40730b77f0929847ea2e6772efdd1"}
Oct 30 20:12:19.445639 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.445616    4823 scope.go:122] "RemoveContainer" containerID="635dad2452b6b4b06f14eb51e27df8beae0404ddccb230204de15dff6d17c1b4"
Oct 30 20:12:19.446008 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:12:19.445987    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635dad2452b6b4b06f14eb51e27df8beae0404ddccb230204de15dff6d17c1b4\": container with ID starting with 635dad2452b6b4b06f14eb51e27df8beae0404ddccb230204de15dff6d17c1b4 not found: ID does not exist" containerID="635dad2452b6b4b06f14eb51e27df8beae0404ddccb230204de15dff6d17c1b4"
Oct 30 20:12:19.446087 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.446016    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635dad2452b6b4b06f14eb51e27df8beae0404ddccb230204de15dff6d17c1b4"} err="failed to get container status \"635dad2452b6b4b06f14eb51e27df8beae0404ddccb230204de15dff6d17c1b4\": rpc error: code = NotFound desc = could not find container \"635dad2452b6b4b06f14eb51e27df8beae0404ddccb230204de15dff6d17c1b4\": container with ID starting with 635dad2452b6b4b06f14eb51e27df8beae0404ddccb230204de15dff6d17c1b4 not found: ID does not exist"
Oct 30 20:12:19.472476 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.472442    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dl6n\" (UniqueName: \"kubernetes.io/projected/fd0d4ac0-27c1-4735-b333-63b4e75afe70-kube-api-access-6dl6n\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:19.489739 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.489715    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"]
Oct 30 20:12:19.495731 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.495706    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a1d885-7d689646db-966sm"]
Oct 30 20:12:19.773289 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.773262    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 20:12:19.877310 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.877275    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-workload-certs\") pod \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") "
Oct 30 20:12:19.877491 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.877334    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-kube-api-access-w89wb\" (UniqueName: \"kubernetes.io/projected/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-kube-api-access-w89wb\") pod \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") "
Oct 30 20:12:19.877491 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.877410    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-data\") pod \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") "
Oct 30 20:12:19.877600 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.877520    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-workload-socket\") pod \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") "
Oct 30 20:12:19.877600 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.877578    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-token\" (UniqueName: \"kubernetes.io/projected/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-token\") pod \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") "
Oct 30 20:12:19.877784 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.877763    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istiod-ca-cert\") pod \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") "
Oct 30 20:12:19.878353 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.877896    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-envoy\") pod \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") "
Oct 30 20:12:19.878353 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.877966    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-credential-socket\") pod \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") "
Oct 30 20:12:19.878353 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.877626    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-workload-certs" pod "7862ef1c-a91c-40f6-abaf-d5eba65b67c5" (UID: "7862ef1c-a91c-40f6-abaf-d5eba65b67c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:19.878353 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.877789    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-data" pod "7862ef1c-a91c-40f6-abaf-d5eba65b67c5" (UID: "7862ef1c-a91c-40f6-abaf-d5eba65b67c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:19.878353 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.877820    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-workload-socket" pod "7862ef1c-a91c-40f6-abaf-d5eba65b67c5" (UID: "7862ef1c-a91c-40f6-abaf-d5eba65b67c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:19.878353 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.878017    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-podinfo\") pod \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\" (UID: \"7862ef1c-a91c-40f6-abaf-d5eba65b67c5\") "
Oct 30 20:12:19.878353 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.878102    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istiod-ca-cert" pod "7862ef1c-a91c-40f6-abaf-d5eba65b67c5" (UID: "7862ef1c-a91c-40f6-abaf-d5eba65b67c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:19.878353 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.878318    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-credential-socket" pod "7862ef1c-a91c-40f6-abaf-d5eba65b67c5" (UID: "7862ef1c-a91c-40f6-abaf-d5eba65b67c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:19.878949 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.878792    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:19.878949 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.878817    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:19.878949 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.878832    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:19.878949 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.878848    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:19.878949 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.878919    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:19.880229 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.880203    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-token" pod "7862ef1c-a91c-40f6-abaf-d5eba65b67c5" (UID: "7862ef1c-a91c-40f6-abaf-d5eba65b67c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:19.880328 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.880293    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-kube-api-access-w89wb" pod "7862ef1c-a91c-40f6-abaf-d5eba65b67c5" (UID: "7862ef1c-a91c-40f6-abaf-d5eba65b67c5"). InnerVolumeSpecName "kube-api-access-w89wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:19.880539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.880521    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-envoy" pod "7862ef1c-a91c-40f6-abaf-d5eba65b67c5" (UID: "7862ef1c-a91c-40f6-abaf-d5eba65b67c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:19.880654 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.880635    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-podinfo" pod "7862ef1c-a91c-40f6-abaf-d5eba65b67c5" (UID: "7862ef1c-a91c-40f6-abaf-d5eba65b67c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:12:19.980851 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.980767    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:19.980851 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.980798    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:19.980851 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.980809    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w89wb\" (UniqueName: \"kubernetes.io/projected/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-kube-api-access-w89wb\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:19.980851 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:19.980818    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7862ef1c-a91c-40f6-abaf-d5eba65b67c5-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:20.284411 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.284316    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd0d4ac0-27c1-4735-b333-63b4e75afe70" path="/var/lib/kubelet/pods/fd0d4ac0-27c1-4735-b333-63b4e75afe70/volumes"
Oct 30 20:12:20.449990 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.449954    4823 generic.go:356] "Generic (PLEG): container finished" podID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" containerID="1e88f27d7c5878928f0ae468e4a054544adf45912e600799202b2d33c8248651" exitCode=0
Oct 30 20:12:20.450466 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.450040    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"
Oct 30 20:12:20.450466 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.450035    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95" event={"ID":"7862ef1c-a91c-40f6-abaf-d5eba65b67c5","Type":"ContainerDied","Data":"1e88f27d7c5878928f0ae468e4a054544adf45912e600799202b2d33c8248651"}
Oct 30 20:12:20.450466 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.450171    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95" event={"ID":"7862ef1c-a91c-40f6-abaf-d5eba65b67c5","Type":"ContainerDied","Data":"c76d7cc73b77119a7d9c8d0be4e5ef41e2fb00f3e0d32845def62983068446f8"}
Oct 30 20:12:20.450466 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.450200    4823 scope.go:122] "RemoveContainer" containerID="1e88f27d7c5878928f0ae468e4a054544adf45912e600799202b2d33c8248651"
Oct 30 20:12:20.461603 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.461517    4823 scope.go:122] "RemoveContainer" containerID="af24e8b669cd122de5bdd2c3880f7b06e4a30aefdfb0f753d9590bfb47ee3c42"
Oct 30 20:12:20.481139 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.481116    4823 scope.go:122] "RemoveContainer" containerID="1e88f27d7c5878928f0ae468e4a054544adf45912e600799202b2d33c8248651"
Oct 30 20:12:20.481512 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:12:20.481481    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e88f27d7c5878928f0ae468e4a054544adf45912e600799202b2d33c8248651\": container with ID starting with 1e88f27d7c5878928f0ae468e4a054544adf45912e600799202b2d33c8248651 not found: ID does not exist" containerID="1e88f27d7c5878928f0ae468e4a054544adf45912e600799202b2d33c8248651"
Oct 30 20:12:20.481605 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.481511    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e88f27d7c5878928f0ae468e4a054544adf45912e600799202b2d33c8248651"} err="failed to get container status \"1e88f27d7c5878928f0ae468e4a054544adf45912e600799202b2d33c8248651\": rpc error: code = NotFound desc = could not find container \"1e88f27d7c5878928f0ae468e4a054544adf45912e600799202b2d33c8248651\": container with ID starting with 1e88f27d7c5878928f0ae468e4a054544adf45912e600799202b2d33c8248651 not found: ID does not exist"
Oct 30 20:12:20.481605 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.481530    4823 scope.go:122] "RemoveContainer" containerID="af24e8b669cd122de5bdd2c3880f7b06e4a30aefdfb0f753d9590bfb47ee3c42"
Oct 30 20:12:20.481823 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:12:20.481794    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af24e8b669cd122de5bdd2c3880f7b06e4a30aefdfb0f753d9590bfb47ee3c42\": container with ID starting with af24e8b669cd122de5bdd2c3880f7b06e4a30aefdfb0f753d9590bfb47ee3c42 not found: ID does not exist" containerID="af24e8b669cd122de5bdd2c3880f7b06e4a30aefdfb0f753d9590bfb47ee3c42"
Oct 30 20:12:20.481909 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.481816    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af24e8b669cd122de5bdd2c3880f7b06e4a30aefdfb0f753d9590bfb47ee3c42"} err="failed to get container status \"af24e8b669cd122de5bdd2c3880f7b06e4a30aefdfb0f753d9590bfb47ee3c42\": rpc error: code = NotFound desc = could not find container \"af24e8b669cd122de5bdd2c3880f7b06e4a30aefdfb0f753d9590bfb47ee3c42\": container with ID starting with af24e8b669cd122de5bdd2c3880f7b06e4a30aefdfb0f753d9590bfb47ee3c42 not found: ID does not exist"
Oct 30 20:12:20.498755 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.498726    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"]
Oct 30 20:12:20.513334 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.513306    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a8de3b-openshift-default-599dcc66b5-v2z95"]
Oct 30 20:12:20.622283 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.622260    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"
Oct 30 20:12:20.697312 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.697272    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/669b522e-bca0-45a3-a6c5-b79a3460414a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/669b522e-bca0-45a3-a6c5-b79a3460414a-maas-api-xks-ca\") pod \"669b522e-bca0-45a3-a6c5-b79a3460414a\" (UID: \"669b522e-bca0-45a3-a6c5-b79a3460414a\") "
Oct 30 20:12:20.697523 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.697412    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/669b522e-bca0-45a3-a6c5-b79a3460414a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/669b522e-bca0-45a3-a6c5-b79a3460414a-maas-api-openshift-ca\") pod \"669b522e-bca0-45a3-a6c5-b79a3460414a\" (UID: \"669b522e-bca0-45a3-a6c5-b79a3460414a\") "
Oct 30 20:12:20.697735 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.697709    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669b522e-bca0-45a3-a6c5-b79a3460414a-kube-api-access-8ctgl\" (UniqueName: \"kubernetes.io/projected/669b522e-bca0-45a3-a6c5-b79a3460414a-kube-api-access-8ctgl\") pod \"669b522e-bca0-45a3-a6c5-b79a3460414a\" (UID: \"669b522e-bca0-45a3-a6c5-b79a3460414a\") "
Oct 30 20:12:20.698086 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.698054    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669b522e-bca0-45a3-a6c5-b79a3460414a-maas-api-openshift-ca" pod "669b522e-bca0-45a3-a6c5-b79a3460414a" (UID: "669b522e-bca0-45a3-a6c5-b79a3460414a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:20.698480 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.698452    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/669b522e-bca0-45a3-a6c5-b79a3460414a-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:20.703195 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.701653    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/669b522e-bca0-45a3-a6c5-b79a3460414a-maas-api-xks-ca" pod "669b522e-bca0-45a3-a6c5-b79a3460414a" (UID: "669b522e-bca0-45a3-a6c5-b79a3460414a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:20.703777 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.703752    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669b522e-bca0-45a3-a6c5-b79a3460414a-kube-api-access-8ctgl" pod "669b522e-bca0-45a3-a6c5-b79a3460414a" (UID: "669b522e-bca0-45a3-a6c5-b79a3460414a"). InnerVolumeSpecName "kube-api-access-8ctgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:20.799534 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.799494    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/669b522e-bca0-45a3-a6c5-b79a3460414a-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:20.799534 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:20.799528    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ctgl\" (UniqueName: \"kubernetes.io/projected/669b522e-bca0-45a3-a6c5-b79a3460414a-kube-api-access-8ctgl\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:21.463423 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:21.463393    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl" event={"ID":"669b522e-bca0-45a3-a6c5-b79a3460414a","Type":"ContainerDied","Data":"548514322c1ee322a242ef7ca7529a931a489fe07b9fc9f5856f0fe01c6c2115"}
Oct 30 20:12:21.463423 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:21.463409    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"
Oct 30 20:12:21.463827 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:21.463441    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="548514322c1ee322a242ef7ca7529a931a489fe07b9fc9f5856f0fe01c6c2115"
Oct 30 20:12:21.936405 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:21.936370    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"]
Oct 30 20:12:21.936638 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:21.936599    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f" podUID="e8828115-a9ef-41dc-bf28-f5dfb3131377" containerName="istio-proxy" containerID="cri-o://2929f0c963c05b52d87d5fd721ee1982e17c0d6757933f1130cb571f6fb4c445" gracePeriod=30
Oct 30 20:12:21.939735 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:21.939703    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"]
Oct 30 20:12:21.939967 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:21.939941    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb" podUID="3c2d0b1a-f650-46ad-bb7f-325ba702a89d" containerName="istio-proxy" containerID="cri-o://21c1795e95e7b4253754d96de06c7ba7d51313fa7e898f2c75526f7a859ce3f1" gracePeriod=30
Oct 30 20:12:22.293295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:22.293205    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" path="/var/lib/kubelet/pods/7862ef1c-a91c-40f6-abaf-d5eba65b67c5/volumes"
Oct 30 20:12:27.259721 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.259688    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 20:12:27.263646 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.263614    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 20:12:27.367757 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.367721    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8828115-a9ef-41dc-bf28-f5dfb3131377-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8828115-a9ef-41dc-bf28-f5dfb3131377-istiod-ca-cert\") pod \"e8828115-a9ef-41dc-bf28-f5dfb3131377\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") "
Oct 30 20:12:27.367950 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.367810    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-workload-certs\") pod \"e8828115-a9ef-41dc-bf28-f5dfb3131377\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") "
Oct 30 20:12:27.367950 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.367841    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-podinfo\") pod \"e8828115-a9ef-41dc-bf28-f5dfb3131377\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") "
Oct 30 20:12:27.368070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.367993    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-credential-socket\") pod \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") "
Oct 30 20:12:27.368070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.368039    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8828115-a9ef-41dc-bf28-f5dfb3131377-kube-api-access-gbbc8\" (UniqueName: \"kubernetes.io/projected/e8828115-a9ef-41dc-bf28-f5dfb3131377-kube-api-access-gbbc8\") pod \"e8828115-a9ef-41dc-bf28-f5dfb3131377\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") "
Oct 30 20:12:27.368647 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.368076    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-workload-socket\") pod \"e8828115-a9ef-41dc-bf28-f5dfb3131377\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") "
Oct 30 20:12:27.368647 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.368106    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-envoy\") pod \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") "
Oct 30 20:12:27.368647 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.368138    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-data\") pod \"e8828115-a9ef-41dc-bf28-f5dfb3131377\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") "
Oct 30 20:12:27.368647 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.368221    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-token\" (UniqueName: \"kubernetes.io/projected/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-token\") pod \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") "
Oct 30 20:12:27.368647 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.368304    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-workload-certs\") pod \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") "
Oct 30 20:12:27.368647 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.368335    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-token\" (UniqueName: \"kubernetes.io/projected/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-token\") pod \"e8828115-a9ef-41dc-bf28-f5dfb3131377\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") "
Oct 30 20:12:27.368647 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.368370    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istiod-ca-cert\") pod \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") "
Oct 30 20:12:27.368647 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.368412    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-podinfo\") pod \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") "
Oct 30 20:12:27.368647 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.368464    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-workload-socket\") pod \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") "
Oct 30 20:12:27.368647 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.368590    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-credential-socket\") pod \"e8828115-a9ef-41dc-bf28-f5dfb3131377\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") "
Oct 30 20:12:27.368647 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.368627    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-envoy\") pod \"e8828115-a9ef-41dc-bf28-f5dfb3131377\" (UID: \"e8828115-a9ef-41dc-bf28-f5dfb3131377\") "
Oct 30 20:12:27.369374 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.368663    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-data\") pod \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") "
Oct 30 20:12:27.369374 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.368704    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-kube-api-access-bx29s\" (UniqueName: \"kubernetes.io/projected/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-kube-api-access-bx29s\") pod \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\" (UID: \"3c2d0b1a-f650-46ad-bb7f-325ba702a89d\") "
Oct 30 20:12:27.370184 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.370144    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-credential-socket" pod "3c2d0b1a-f650-46ad-bb7f-325ba702a89d" (UID: "3c2d0b1a-f650-46ad-bb7f-325ba702a89d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:27.370400 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.370375    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-workload-certs" pod "3c2d0b1a-f650-46ad-bb7f-325ba702a89d" (UID: "3c2d0b1a-f650-46ad-bb7f-325ba702a89d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:27.370855 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.370520    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8828115-a9ef-41dc-bf28-f5dfb3131377-istiod-ca-cert" pod "e8828115-a9ef-41dc-bf28-f5dfb3131377" (UID: "e8828115-a9ef-41dc-bf28-f5dfb3131377"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:27.371005 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.370795    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-workload-certs" pod "e8828115-a9ef-41dc-bf28-f5dfb3131377" (UID: "e8828115-a9ef-41dc-bf28-f5dfb3131377"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:27.371279 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.371261    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-data" pod "3c2d0b1a-f650-46ad-bb7f-325ba702a89d" (UID: "3c2d0b1a-f650-46ad-bb7f-325ba702a89d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:27.371661 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.371645    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istiod-ca-cert" pod "3c2d0b1a-f650-46ad-bb7f-325ba702a89d" (UID: "3c2d0b1a-f650-46ad-bb7f-325ba702a89d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:27.372038 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.372018    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-workload-socket" pod "e8828115-a9ef-41dc-bf28-f5dfb3131377" (UID: "e8828115-a9ef-41dc-bf28-f5dfb3131377"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:27.372974 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.372640    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-token" pod "3c2d0b1a-f650-46ad-bb7f-325ba702a89d" (UID: "3c2d0b1a-f650-46ad-bb7f-325ba702a89d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:27.373274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.373252    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-credential-socket" pod "e8828115-a9ef-41dc-bf28-f5dfb3131377" (UID: "e8828115-a9ef-41dc-bf28-f5dfb3131377"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:27.373633 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.373440    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-workload-socket" pod "3c2d0b1a-f650-46ad-bb7f-325ba702a89d" (UID: "3c2d0b1a-f650-46ad-bb7f-325ba702a89d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:27.374557 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.374530    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-data" pod "e8828115-a9ef-41dc-bf28-f5dfb3131377" (UID: "e8828115-a9ef-41dc-bf28-f5dfb3131377"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:27.375609 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.375585    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-kube-api-access-bx29s" pod "3c2d0b1a-f650-46ad-bb7f-325ba702a89d" (UID: "3c2d0b1a-f650-46ad-bb7f-325ba702a89d"). InnerVolumeSpecName "kube-api-access-bx29s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:27.377247 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.377214    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-envoy" pod "3c2d0b1a-f650-46ad-bb7f-325ba702a89d" (UID: "3c2d0b1a-f650-46ad-bb7f-325ba702a89d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:27.379176 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.379157    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-podinfo" pod "3c2d0b1a-f650-46ad-bb7f-325ba702a89d" (UID: "3c2d0b1a-f650-46ad-bb7f-325ba702a89d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:12:27.379363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.379345    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-token" pod "e8828115-a9ef-41dc-bf28-f5dfb3131377" (UID: "e8828115-a9ef-41dc-bf28-f5dfb3131377"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:27.380311 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.380112    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-podinfo" pod "e8828115-a9ef-41dc-bf28-f5dfb3131377" (UID: "e8828115-a9ef-41dc-bf28-f5dfb3131377"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:12:27.380402 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.380329    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-envoy" pod "e8828115-a9ef-41dc-bf28-f5dfb3131377" (UID: "e8828115-a9ef-41dc-bf28-f5dfb3131377"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:27.381053 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.381031    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8828115-a9ef-41dc-bf28-f5dfb3131377-kube-api-access-gbbc8" pod "e8828115-a9ef-41dc-bf28-f5dfb3131377" (UID: "e8828115-a9ef-41dc-bf28-f5dfb3131377"). InnerVolumeSpecName "kube-api-access-gbbc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:27.471113 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471080    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471113 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471109    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471113 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471119    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471128    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx29s\" (UniqueName: \"kubernetes.io/projected/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-kube-api-access-bx29s\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471139    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8828115-a9ef-41dc-bf28-f5dfb3131377-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471147    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471156    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471175    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471183    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbbc8\" (UniqueName: \"kubernetes.io/projected/e8828115-a9ef-41dc-bf28-f5dfb3131377-kube-api-access-gbbc8\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471192    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471200    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471208    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471216    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471224    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471232    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8828115-a9ef-41dc-bf28-f5dfb3131377-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471241    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471249    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.471351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.471257    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d0b1a-f650-46ad-bb7f-325ba702a89d-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:27.659262 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.659217    4823 generic.go:356] "Generic (PLEG): container finished" podID="e8828115-a9ef-41dc-bf28-f5dfb3131377" containerID="2929f0c963c05b52d87d5fd721ee1982e17c0d6757933f1130cb571f6fb4c445" exitCode=0
Oct 30 20:12:27.659936 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.659570    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"
Oct 30 20:12:27.666048 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.659594    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f" event={"ID":"e8828115-a9ef-41dc-bf28-f5dfb3131377","Type":"ContainerDied","Data":"2929f0c963c05b52d87d5fd721ee1982e17c0d6757933f1130cb571f6fb4c445"}
Oct 30 20:12:27.666610 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.666584    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f" event={"ID":"e8828115-a9ef-41dc-bf28-f5dfb3131377","Type":"ContainerDied","Data":"161f8146c8407bbef8511f0fc8b429582baac55bdac0137096c39ba88033fe31"}
Oct 30 20:12:27.666809 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.666794    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb" event={"ID":"3c2d0b1a-f650-46ad-bb7f-325ba702a89d","Type":"ContainerDied","Data":"21c1795e95e7b4253754d96de06c7ba7d51313fa7e898f2c75526f7a859ce3f1"}
Oct 30 20:12:27.666952 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.666752    4823 scope.go:122] "RemoveContainer" containerID="2929f0c963c05b52d87d5fd721ee1982e17c0d6757933f1130cb571f6fb4c445"
Oct 30 20:12:27.667149 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.664943    4823 generic.go:356] "Generic (PLEG): container finished" podID="3c2d0b1a-f650-46ad-bb7f-325ba702a89d" containerID="21c1795e95e7b4253754d96de06c7ba7d51313fa7e898f2c75526f7a859ce3f1" exitCode=0
Oct 30 20:12:27.668806 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.668788    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb" event={"ID":"3c2d0b1a-f650-46ad-bb7f-325ba702a89d","Type":"ContainerDied","Data":"419630ef5148acc98ca587b4bb0f832dc414abc435adeb64388716ccb2b14222"}
Oct 30 20:12:27.668996 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.665012    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"
Oct 30 20:12:27.753882 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.743335    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"]
Oct 30 20:12:27.753882 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.745739    4823 scope.go:122] "RemoveContainer" containerID="2929f0c963c05b52d87d5fd721ee1982e17c0d6757933f1130cb571f6fb4c445"
Oct 30 20:12:27.753882 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:12:27.746172    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2929f0c963c05b52d87d5fd721ee1982e17c0d6757933f1130cb571f6fb4c445\": container with ID starting with 2929f0c963c05b52d87d5fd721ee1982e17c0d6757933f1130cb571f6fb4c445 not found: ID does not exist" containerID="2929f0c963c05b52d87d5fd721ee1982e17c0d6757933f1130cb571f6fb4c445"
Oct 30 20:12:27.753882 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.746248    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2929f0c963c05b52d87d5fd721ee1982e17c0d6757933f1130cb571f6fb4c445"} err="failed to get container status \"2929f0c963c05b52d87d5fd721ee1982e17c0d6757933f1130cb571f6fb4c445\": rpc error: code = NotFound desc = could not find container \"2929f0c963c05b52d87d5fd721ee1982e17c0d6757933f1130cb571f6fb4c445\": container with ID starting with 2929f0c963c05b52d87d5fd721ee1982e17c0d6757933f1130cb571f6fb4c445 not found: ID does not exist"
Oct 30 20:12:27.753882 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.746267    4823 scope.go:122] "RemoveContainer" containerID="21c1795e95e7b4253754d96de06c7ba7d51313fa7e898f2c75526f7a859ce3f1"
Oct 30 20:12:27.753882 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.750465    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-tbb6f"]
Oct 30 20:12:27.764093 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.764013    4823 scope.go:122] "RemoveContainer" containerID="21c1795e95e7b4253754d96de06c7ba7d51313fa7e898f2c75526f7a859ce3f1"
Oct 30 20:12:27.764550 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.764459    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"]
Oct 30 20:12:27.764638 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:12:27.764596    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c1795e95e7b4253754d96de06c7ba7d51313fa7e898f2c75526f7a859ce3f1\": container with ID starting with 21c1795e95e7b4253754d96de06c7ba7d51313fa7e898f2c75526f7a859ce3f1 not found: ID does not exist" containerID="21c1795e95e7b4253754d96de06c7ba7d51313fa7e898f2c75526f7a859ce3f1"
Oct 30 20:12:27.764638 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.764623    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c1795e95e7b4253754d96de06c7ba7d51313fa7e898f2c75526f7a859ce3f1"} err="failed to get container status \"21c1795e95e7b4253754d96de06c7ba7d51313fa7e898f2c75526f7a859ce3f1\": rpc error: code = NotFound desc = could not find container \"21c1795e95e7b4253754d96de06c7ba7d51313fa7e898f2c75526f7a859ce3f1\": container with ID starting with 21c1795e95e7b4253754d96de06c7ba7d51313fa7e898f2c75526f7a859ce3f1 not found: ID does not exist"
Oct 30 20:12:27.771783 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:27.771752    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-bmtgb"]
Oct 30 20:12:28.283232 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:28.283154    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c2d0b1a-f650-46ad-bb7f-325ba702a89d" path="/var/lib/kubelet/pods/3c2d0b1a-f650-46ad-bb7f-325ba702a89d/volumes"
Oct 30 20:12:28.284570 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:28.284542    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8828115-a9ef-41dc-bf28-f5dfb3131377" path="/var/lib/kubelet/pods/e8828115-a9ef-41dc-bf28-f5dfb3131377/volumes"
Oct 30 20:12:28.794875 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:28.794827    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"]
Oct 30 20:12:28.795374 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:28.795342    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw" podUID="3c4543ad-ea76-4398-b7f2-7d7a32132821" containerName="istio-proxy" containerID="cri-o://d6eb2da20b5918925e4f098d2183523084dc4ec1ad8a043136ca04ede720116a" gracePeriod=30
Oct 30 20:12:28.898053 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:28.897827    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"]
Oct 30 20:12:28.911448 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:28.911416    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f5d176-wbmtq"]
Oct 30 20:12:29.642423 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.642392    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"]
Oct 30 20:12:29.643302 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643277    4823 cpu_manager.go:414] "containerMap: removing container" podUID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" containerName="istio-proxy"
Oct 30 20:12:29.643302 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643302    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" containerName="istio-proxy"
Oct 30 20:12:29.643475 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643322    4823 cpu_manager.go:414] "containerMap: removing container" podUID="669b522e-bca0-45a3-a6c5-b79a3460414a" containerName="revoke-keys"
Oct 30 20:12:29.643475 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643329    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="669b522e-bca0-45a3-a6c5-b79a3460414a" containerName="revoke-keys"
Oct 30 20:12:29.643475 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643359    4823 cpu_manager.go:414] "containerMap: removing container" podUID="3c2d0b1a-f650-46ad-bb7f-325ba702a89d" containerName="istio-proxy"
Oct 30 20:12:29.643475 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643367    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c2d0b1a-f650-46ad-bb7f-325ba702a89d" containerName="istio-proxy"
Oct 30 20:12:29.643475 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643393    4823 cpu_manager.go:414] "containerMap: removing container" podUID="712538e4-19f3-4e54-94ab-d6b435581839" containerName="revoke-keys"
Oct 30 20:12:29.643475 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643400    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712538e4-19f3-4e54-94ab-d6b435581839" containerName="revoke-keys"
Oct 30 20:12:29.643475 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643429    4823 cpu_manager.go:414] "containerMap: removing container" podUID="e8828115-a9ef-41dc-bf28-f5dfb3131377" containerName="istio-proxy"
Oct 30 20:12:29.643475 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643437    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8828115-a9ef-41dc-bf28-f5dfb3131377" containerName="istio-proxy"
Oct 30 20:12:29.643475 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643452    4823 cpu_manager.go:414] "containerMap: removing container" podUID="fd0d4ac0-27c1-4735-b333-63b4e75afe70" containerName="payload-processing"
Oct 30 20:12:29.643475 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643460    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0d4ac0-27c1-4735-b333-63b4e75afe70" containerName="payload-processing"
Oct 30 20:12:29.644022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643609    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="712538e4-19f3-4e54-94ab-d6b435581839" containerName="revoke-keys"
Oct 30 20:12:29.644022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643627    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8828115-a9ef-41dc-bf28-f5dfb3131377" containerName="istio-proxy"
Oct 30 20:12:29.644022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643638    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" containerName="istio-proxy"
Oct 30 20:12:29.644022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643646    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0d4ac0-27c1-4735-b333-63b4e75afe70" containerName="payload-processing"
Oct 30 20:12:29.644022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643660    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" containerName="istio-proxy"
Oct 30 20:12:29.644022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643669    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="669b522e-bca0-45a3-a6c5-b79a3460414a" containerName="revoke-keys"
Oct 30 20:12:29.644022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.643682    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c2d0b1a-f650-46ad-bb7f-325ba702a89d" containerName="istio-proxy"
Oct 30 20:12:29.649328 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.649305    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"
Oct 30 20:12:29.659289 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.659263    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"]
Oct 30 20:12:29.659831 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.659816    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:12:29.659994 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.659978    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:12:29.722329 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.722297    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8bbafa-1167-49d4-a51b-a90801d5d66b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq\" (UID: \"4e8bbafa-1167-49d4-a51b-a90801d5d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"
Oct 30 20:12:29.722516 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.722348    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8bbafa-1167-49d4-a51b-a90801d5d66b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq\" (UID: \"4e8bbafa-1167-49d4-a51b-a90801d5d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"
Oct 30 20:12:29.722516 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.722462    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwbhr\" (UniqueName: \"kubernetes.io/projected/4e8bbafa-1167-49d4-a51b-a90801d5d66b-kube-api-access-nwbhr\") pod \"maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq\" (UID: \"4e8bbafa-1167-49d4-a51b-a90801d5d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"
Oct 30 20:12:29.755361 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.755330    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:12:29.755361 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.755367    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:12:29.823608 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.823572    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8bbafa-1167-49d4-a51b-a90801d5d66b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq\" (UID: \"4e8bbafa-1167-49d4-a51b-a90801d5d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"
Oct 30 20:12:29.823822 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.823660    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8bbafa-1167-49d4-a51b-a90801d5d66b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq\" (UID: \"4e8bbafa-1167-49d4-a51b-a90801d5d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"
Oct 30 20:12:29.823822 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.823733    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwbhr\" (UniqueName: \"kubernetes.io/projected/4e8bbafa-1167-49d4-a51b-a90801d5d66b-kube-api-access-nwbhr\") pod \"maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq\" (UID: \"4e8bbafa-1167-49d4-a51b-a90801d5d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"
Oct 30 20:12:29.824417 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.824388    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8bbafa-1167-49d4-a51b-a90801d5d66b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq\" (UID: \"4e8bbafa-1167-49d4-a51b-a90801d5d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"
Oct 30 20:12:29.826525 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.826496    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8bbafa-1167-49d4-a51b-a90801d5d66b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq\" (UID: \"4e8bbafa-1167-49d4-a51b-a90801d5d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"
Oct 30 20:12:29.841358 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.841331    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwbhr\" (UniqueName: \"kubernetes.io/projected/4e8bbafa-1167-49d4-a51b-a90801d5d66b-kube-api-access-nwbhr\") pod \"maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq\" (UID: \"4e8bbafa-1167-49d4-a51b-a90801d5d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"
Oct 30 20:12:29.967048 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:29.966856    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"
Oct 30 20:12:30.254904 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:30.254857    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"]
Oct 30 20:12:30.255052 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:12:30.254959    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8bbafa_1167_49d4_a51b_a90801d5d66b.slice/crio-b59916405ed1fb265270a88d2db763cb4f8a41699f64b4d08dad40d33cd035e1 WatchSource:0}: Error finding container b59916405ed1fb265270a88d2db763cb4f8a41699f64b4d08dad40d33cd035e1: Status 404 returned error can't find the container with id b59916405ed1fb265270a88d2db763cb4f8a41699f64b4d08dad40d33cd035e1
Oct 30 20:12:30.281994 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:30.281971    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="712538e4-19f3-4e54-94ab-d6b435581839" path="/var/lib/kubelet/pods/712538e4-19f3-4e54-94ab-d6b435581839/volumes"
Oct 30 20:12:30.705954 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:30.705921    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq" event={"ID":"4e8bbafa-1167-49d4-a51b-a90801d5d66b","Type":"ContainerStarted","Data":"e1bc0a989637882fe0f9795fe20eca35d4268bcc7602b0c70b092d8d5165cdd6"}
Oct 30 20:12:30.706393 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:30.705976    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq" event={"ID":"4e8bbafa-1167-49d4-a51b-a90801d5d66b","Type":"ContainerStarted","Data":"b59916405ed1fb265270a88d2db763cb4f8a41699f64b4d08dad40d33cd035e1"}
Oct 30 20:12:30.734120 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:30.731905    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq" podStartSLOduration=1.731886714 podStartE2EDuration="1.731886714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:30.726646149 +0000 UTC m=+2550.619260473" watchObservedRunningTime="2026-10-30 20:12:30.731886714 +0000 UTC m=+2550.624501039"
Oct 30 20:12:31.407649 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.407591    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main" containerID="cri-o://0db201eb8100ded6ef7992935a6e4c640a46ca6c230652e33e181277c19c03ab" gracePeriod=45
Oct 30 20:12:31.444159 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.444119    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main" containerID="cri-o://ce2ef19d9aa332e404fcc4ba4521dac247be996219436aeb0a2fabe7169c09b1" gracePeriod=45
Oct 30 20:12:31.722340 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.722254    4823 generic.go:356] "Generic (PLEG): container finished" podID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerID="ce2ef19d9aa332e404fcc4ba4521dac247be996219436aeb0a2fabe7169c09b1" exitCode=0
Oct 30 20:12:31.722747 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.722330    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" event={"ID":"60e95f14-dd0c-47d5-92ef-5acc29210a5d","Type":"ContainerDied","Data":"ce2ef19d9aa332e404fcc4ba4521dac247be996219436aeb0a2fabe7169c09b1"}
Oct 30 20:12:31.724451 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.724391    4823 generic.go:356] "Generic (PLEG): container finished" podID="144ed52c-29ff-42f9-8695-701aef451ce5" containerID="0db201eb8100ded6ef7992935a6e4c640a46ca6c230652e33e181277c19c03ab" exitCode=0
Oct 30 20:12:31.724626 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.724451    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" event={"ID":"144ed52c-29ff-42f9-8695-701aef451ce5","Type":"ContainerDied","Data":"0db201eb8100ded6ef7992935a6e4c640a46ca6c230652e33e181277c19c03ab"}
Oct 30 20:12:31.726822 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.726580    4823 generic.go:356] "Generic (PLEG): container finished" podID="4e8bbafa-1167-49d4-a51b-a90801d5d66b" containerID="e1bc0a989637882fe0f9795fe20eca35d4268bcc7602b0c70b092d8d5165cdd6" exitCode=0
Oct 30 20:12:31.726822 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.726688    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq" event={"ID":"4e8bbafa-1167-49d4-a51b-a90801d5d66b","Type":"ContainerDied","Data":"e1bc0a989637882fe0f9795fe20eca35d4268bcc7602b0c70b092d8d5165cdd6"}
Oct 30 20:12:31.748260 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.748238    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 20:12:31.757059 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.757036    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 20:12:31.858700 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.858669    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-model-cache\") pod \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") "
Oct 30 20:12:31.858919 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.858711    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-dshm\") pod \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") "
Oct 30 20:12:31.858919 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.858735    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-model-cache\") pod \"144ed52c-29ff-42f9-8695-701aef451ce5\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") "
Oct 30 20:12:31.858919 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.858766    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-home\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-home\") pod \"144ed52c-29ff-42f9-8695-701aef451ce5\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") "
Oct 30 20:12:31.858919 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.858814    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/144ed52c-29ff-42f9-8695-701aef451ce5-tls-certs\" (UniqueName: \"kubernetes.io/secret/144ed52c-29ff-42f9-8695-701aef451ce5-tls-certs\") pod \"144ed52c-29ff-42f9-8695-701aef451ce5\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") "
Oct 30 20:12:31.858919 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.858847    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e95f14-dd0c-47d5-92ef-5acc29210a5d-kube-api-access-8t9tz\" (UniqueName: \"kubernetes.io/projected/60e95f14-dd0c-47d5-92ef-5acc29210a5d-kube-api-access-8t9tz\") pod \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") "
Oct 30 20:12:31.858919 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.858898    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-tmp-dir\") pod \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") "
Oct 30 20:12:31.858919 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.858916    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-tmp-dir\") pod \"144ed52c-29ff-42f9-8695-701aef451ce5\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") "
Oct 30 20:12:31.859389 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.858942    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-dshm\") pod \"144ed52c-29ff-42f9-8695-701aef451ce5\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") "
Oct 30 20:12:31.859389 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.859006    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-model-cache" pod "60e95f14-dd0c-47d5-92ef-5acc29210a5d" (UID: "60e95f14-dd0c-47d5-92ef-5acc29210a5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:31.859389 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.859046    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e95f14-dd0c-47d5-92ef-5acc29210a5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/60e95f14-dd0c-47d5-92ef-5acc29210a5d-tls-certs\") pod \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") "
Oct 30 20:12:31.859389 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.859073    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-model-cache" pod "144ed52c-29ff-42f9-8695-701aef451ce5" (UID: "144ed52c-29ff-42f9-8695-701aef451ce5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:31.859389 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.859107    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144ed52c-29ff-42f9-8695-701aef451ce5-kube-api-access-7ghtq\" (UniqueName: \"kubernetes.io/projected/144ed52c-29ff-42f9-8695-701aef451ce5-kube-api-access-7ghtq\") pod \"144ed52c-29ff-42f9-8695-701aef451ce5\" (UID: \"144ed52c-29ff-42f9-8695-701aef451ce5\") "
Oct 30 20:12:31.859389 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.859248    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-home\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-home\") pod \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\" (UID: \"60e95f14-dd0c-47d5-92ef-5acc29210a5d\") "
Oct 30 20:12:31.859389 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.859283    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-tmp-dir" pod "144ed52c-29ff-42f9-8695-701aef451ce5" (UID: "144ed52c-29ff-42f9-8695-701aef451ce5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:31.859777 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.859566    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-tmp-dir" pod "60e95f14-dd0c-47d5-92ef-5acc29210a5d" (UID: "60e95f14-dd0c-47d5-92ef-5acc29210a5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:31.859844 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.859775    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-home" pod "60e95f14-dd0c-47d5-92ef-5acc29210a5d" (UID: "60e95f14-dd0c-47d5-92ef-5acc29210a5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:31.859844 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.859811    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-home" pod "144ed52c-29ff-42f9-8695-701aef451ce5" (UID: "144ed52c-29ff-42f9-8695-701aef451ce5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:31.860121 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.860098    4823 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-home\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:31.860220 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.860129    4823 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-model-cache\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:31.860220 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.860146    4823 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-model-cache\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:31.860220 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.860161    4823 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-home\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:31.860220 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.860175    4823 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-tmp-dir\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:31.860220 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.860189    4823 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-tmp-dir\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:31.861543 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.861522    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e95f14-dd0c-47d5-92ef-5acc29210a5d-tls-certs" pod "60e95f14-dd0c-47d5-92ef-5acc29210a5d" (UID: "60e95f14-dd0c-47d5-92ef-5acc29210a5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:31.861745 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.861726    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e95f14-dd0c-47d5-92ef-5acc29210a5d-kube-api-access-8t9tz" pod "60e95f14-dd0c-47d5-92ef-5acc29210a5d" (UID: "60e95f14-dd0c-47d5-92ef-5acc29210a5d"). InnerVolumeSpecName "kube-api-access-8t9tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:31.862078 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.862050    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-dshm" pod "60e95f14-dd0c-47d5-92ef-5acc29210a5d" (UID: "60e95f14-dd0c-47d5-92ef-5acc29210a5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:31.862174 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.862137    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144ed52c-29ff-42f9-8695-701aef451ce5-tls-certs" pod "144ed52c-29ff-42f9-8695-701aef451ce5" (UID: "144ed52c-29ff-42f9-8695-701aef451ce5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:31.862262 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.862243    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144ed52c-29ff-42f9-8695-701aef451ce5-kube-api-access-7ghtq" pod "144ed52c-29ff-42f9-8695-701aef451ce5" (UID: "144ed52c-29ff-42f9-8695-701aef451ce5"). InnerVolumeSpecName "kube-api-access-7ghtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:31.862395 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.862381    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-dshm" pod "144ed52c-29ff-42f9-8695-701aef451ce5" (UID: "144ed52c-29ff-42f9-8695-701aef451ce5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:31.961485 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.961451    4823 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e95f14-dd0c-47d5-92ef-5acc29210a5d-tls-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:31.961485 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.961484    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ghtq\" (UniqueName: \"kubernetes.io/projected/144ed52c-29ff-42f9-8695-701aef451ce5-kube-api-access-7ghtq\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:31.961485 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.961494    4823 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e95f14-dd0c-47d5-92ef-5acc29210a5d-dshm\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:31.961718 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.961502    4823 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144ed52c-29ff-42f9-8695-701aef451ce5-tls-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:31.961718 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.961511    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t9tz\" (UniqueName: \"kubernetes.io/projected/60e95f14-dd0c-47d5-92ef-5acc29210a5d-kube-api-access-8t9tz\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:31.961718 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:31.961522    4823 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144ed52c-29ff-42f9-8695-701aef451ce5-dshm\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:32.019469 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:12:32.019376    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:12:32.059371 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.059338    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"]
Oct 30 20:12:32.060094 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.060073    4823 cpu_manager.go:414] "containerMap: removing container" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main"
Oct 30 20:12:32.060186 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.060097    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main"
Oct 30 20:12:32.060186 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.060179    4823 cpu_manager.go:414] "containerMap: removing container" podUID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" containerName="istio-proxy"
Oct 30 20:12:32.060276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.060188    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7862ef1c-a91c-40f6-abaf-d5eba65b67c5" containerName="istio-proxy"
Oct 30 20:12:32.060276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.060203    4823 cpu_manager.go:414] "containerMap: removing container" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main"
Oct 30 20:12:32.060276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.060211    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main"
Oct 30 20:12:32.060446 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.060350    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" containerName="main"
Oct 30 20:12:32.060446 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.060363    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" containerName="main"
Oct 30 20:12:32.065057 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.065036    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"
Oct 30 20:12:32.079360 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.078604    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"]
Oct 30 20:12:32.165088 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.165045    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03784eb-84ppm\" (UID: \"432a6a67-fb8e-46da-aa1d-61aa0f4721a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"
Oct 30 20:12:32.165270 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.165110    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03784eb-84ppm\" (UID: \"432a6a67-fb8e-46da-aa1d-61aa0f4721a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"
Oct 30 20:12:32.165320 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.165287    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pswk7\" (UniqueName: \"kubernetes.io/projected/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-kube-api-access-pswk7\") pod \"maas-api-revoke-sub-f03784eb-84ppm\" (UID: \"432a6a67-fb8e-46da-aa1d-61aa0f4721a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"
Oct 30 20:12:32.266790 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.266749    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pswk7\" (UniqueName: \"kubernetes.io/projected/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-kube-api-access-pswk7\") pod \"maas-api-revoke-sub-f03784eb-84ppm\" (UID: \"432a6a67-fb8e-46da-aa1d-61aa0f4721a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"
Oct 30 20:12:32.267031 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.266949    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03784eb-84ppm\" (UID: \"432a6a67-fb8e-46da-aa1d-61aa0f4721a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"
Oct 30 20:12:32.267031 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.267009    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03784eb-84ppm\" (UID: \"432a6a67-fb8e-46da-aa1d-61aa0f4721a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"
Oct 30 20:12:32.267777 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.267756    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03784eb-84ppm\" (UID: \"432a6a67-fb8e-46da-aa1d-61aa0f4721a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"
Oct 30 20:12:32.269407 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.269384    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03784eb-84ppm\" (UID: \"432a6a67-fb8e-46da-aa1d-61aa0f4721a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"
Oct 30 20:12:32.284898 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.284817    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pswk7\" (UniqueName: \"kubernetes.io/projected/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-kube-api-access-pswk7\") pod \"maas-api-revoke-sub-f03784eb-84ppm\" (UID: \"432a6a67-fb8e-46da-aa1d-61aa0f4721a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"
Oct 30 20:12:32.387057 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.387029    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"
Oct 30 20:12:32.693738 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.693709    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"]
Oct 30 20:12:32.695456 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:12:32.695429    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432a6a67_fb8e_46da_aa1d_61aa0f4721a1.slice/crio-11fbcb2136892dfece5c8d70258b9f90295bd757ed51db3c00ac1cade9fb36aa WatchSource:0}: Error finding container 11fbcb2136892dfece5c8d70258b9f90295bd757ed51db3c00ac1cade9fb36aa: Status 404 returned error can't find the container with id 11fbcb2136892dfece5c8d70258b9f90295bd757ed51db3c00ac1cade9fb36aa
Oct 30 20:12:32.737604 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.737580    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"
Oct 30 20:12:32.737998 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.737609    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6" event={"ID":"60e95f14-dd0c-47d5-92ef-5acc29210a5d","Type":"ContainerDied","Data":"21cf894df2e6d2f70b09edbda26ec62603ec2adc01a967092dcfb03330e9cda3"}
Oct 30 20:12:32.737998 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.737655    4823 scope.go:122] "RemoveContainer" containerID="ce2ef19d9aa332e404fcc4ba4521dac247be996219436aeb0a2fabe7169c09b1"
Oct 30 20:12:32.739285 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.739262    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"
Oct 30 20:12:32.739377 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.739289    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg" event={"ID":"144ed52c-29ff-42f9-8695-701aef451ce5","Type":"ContainerDied","Data":"9f17e44d1433908d7497f3dbbf453e4719b596cde0eea9cc2ce0f6869348dc2e"}
Oct 30 20:12:32.740857 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.740832    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm" event={"ID":"432a6a67-fb8e-46da-aa1d-61aa0f4721a1","Type":"ContainerStarted","Data":"11fbcb2136892dfece5c8d70258b9f90295bd757ed51db3c00ac1cade9fb36aa"}
Oct 30 20:12:32.749126 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.749104    4823 scope.go:122] "RemoveContainer" containerID="0db201eb8100ded6ef7992935a6e4c640a46ca6c230652e33e181277c19c03ab"
Oct 30 20:12:32.770181 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.770152    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"]
Oct 30 20:12:32.775654 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.775624    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/simulator-w4-3f57c5-kserve-7797496b8-4k7h6"]
Oct 30 20:12:32.785943 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.785918    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"]
Oct 30 20:12:32.797200 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.797175    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/distinct-2-w4-3f57c5-kserve-7dbf4858cf-kmbdg"]
Oct 30 20:12:32.915319 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.915296    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"
Oct 30 20:12:32.979018 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.978940    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8bbafa-1167-49d4-a51b-a90801d5d66b-kube-api-access-nwbhr\" (UniqueName: \"kubernetes.io/projected/4e8bbafa-1167-49d4-a51b-a90801d5d66b-kube-api-access-nwbhr\") pod \"4e8bbafa-1167-49d4-a51b-a90801d5d66b\" (UID: \"4e8bbafa-1167-49d4-a51b-a90801d5d66b\") "
Oct 30 20:12:32.979170 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.979052    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e8bbafa-1167-49d4-a51b-a90801d5d66b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8bbafa-1167-49d4-a51b-a90801d5d66b-maas-api-xks-ca\") pod \"4e8bbafa-1167-49d4-a51b-a90801d5d66b\" (UID: \"4e8bbafa-1167-49d4-a51b-a90801d5d66b\") "
Oct 30 20:12:32.979170 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.979123    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e8bbafa-1167-49d4-a51b-a90801d5d66b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8bbafa-1167-49d4-a51b-a90801d5d66b-maas-api-openshift-ca\") pod \"4e8bbafa-1167-49d4-a51b-a90801d5d66b\" (UID: \"4e8bbafa-1167-49d4-a51b-a90801d5d66b\") "
Oct 30 20:12:32.979515 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.979482    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8bbafa-1167-49d4-a51b-a90801d5d66b-maas-api-openshift-ca" pod "4e8bbafa-1167-49d4-a51b-a90801d5d66b" (UID: "4e8bbafa-1167-49d4-a51b-a90801d5d66b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:32.979744 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.979719    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8bbafa-1167-49d4-a51b-a90801d5d66b-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:32.981443 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.981416    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e8bbafa-1167-49d4-a51b-a90801d5d66b-maas-api-xks-ca" pod "4e8bbafa-1167-49d4-a51b-a90801d5d66b" (UID: "4e8bbafa-1167-49d4-a51b-a90801d5d66b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:32.981539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:32.981437    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8bbafa-1167-49d4-a51b-a90801d5d66b-kube-api-access-nwbhr" pod "4e8bbafa-1167-49d4-a51b-a90801d5d66b" (UID: "4e8bbafa-1167-49d4-a51b-a90801d5d66b"). InnerVolumeSpecName "kube-api-access-nwbhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:33.081536 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:33.081510    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwbhr\" (UniqueName: \"kubernetes.io/projected/4e8bbafa-1167-49d4-a51b-a90801d5d66b-kube-api-access-nwbhr\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:33.081690 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:33.081539    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8bbafa-1167-49d4-a51b-a90801d5d66b-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:33.751930 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:33.751882    4823 generic.go:356] "Generic (PLEG): container finished" podID="432a6a67-fb8e-46da-aa1d-61aa0f4721a1" containerID="6a01f7d35bcccddde6bae38d00982232b557af285240438d43bc462736009a58" exitCode=0
Oct 30 20:12:33.752408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:33.751959    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm" event={"ID":"432a6a67-fb8e-46da-aa1d-61aa0f4721a1","Type":"ContainerDied","Data":"6a01f7d35bcccddde6bae38d00982232b557af285240438d43bc462736009a58"}
Oct 30 20:12:33.753501 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:33.753472    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq" event={"ID":"4e8bbafa-1167-49d4-a51b-a90801d5d66b","Type":"ContainerDied","Data":"b59916405ed1fb265270a88d2db763cb4f8a41699f64b4d08dad40d33cd035e1"}
Oct 30 20:12:33.753660 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:33.753511    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b59916405ed1fb265270a88d2db763cb4f8a41699f64b4d08dad40d33cd035e1"
Oct 30 20:12:33.753660 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:33.753534    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"
Oct 30 20:12:34.073972 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.073945    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:12:34.195488 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.195408    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-token\" (UniqueName: \"kubernetes.io/projected/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-token\") pod \"3c4543ad-ea76-4398-b7f2-7d7a32132821\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") "
Oct 30 20:12:34.195644 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.195519    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4543ad-ea76-4398-b7f2-7d7a32132821-kube-api-access-4swls\" (UniqueName: \"kubernetes.io/projected/3c4543ad-ea76-4398-b7f2-7d7a32132821-kube-api-access-4swls\") pod \"3c4543ad-ea76-4398-b7f2-7d7a32132821\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") "
Oct 30 20:12:34.195644 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.195613    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-data\") pod \"3c4543ad-ea76-4398-b7f2-7d7a32132821\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") "
Oct 30 20:12:34.195721 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.195684    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-podinfo\") pod \"3c4543ad-ea76-4398-b7f2-7d7a32132821\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") "
Oct 30 20:12:34.195756 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.195728    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-workload-certs\") pod \"3c4543ad-ea76-4398-b7f2-7d7a32132821\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") "
Oct 30 20:12:34.195806 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.195754    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-credential-socket\") pod \"3c4543ad-ea76-4398-b7f2-7d7a32132821\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") "
Oct 30 20:12:34.195806 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.195803    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c4543ad-ea76-4398-b7f2-7d7a32132821-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4543ad-ea76-4398-b7f2-7d7a32132821-istiod-ca-cert\") pod \"3c4543ad-ea76-4398-b7f2-7d7a32132821\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") "
Oct 30 20:12:34.195967 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.195820    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-workload-socket\") pod \"3c4543ad-ea76-4398-b7f2-7d7a32132821\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") "
Oct 30 20:12:34.195967 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.195903    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-envoy\") pod \"3c4543ad-ea76-4398-b7f2-7d7a32132821\" (UID: \"3c4543ad-ea76-4398-b7f2-7d7a32132821\") "
Oct 30 20:12:34.195967 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.195921    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-data" pod "3c4543ad-ea76-4398-b7f2-7d7a32132821" (UID: "3c4543ad-ea76-4398-b7f2-7d7a32132821"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:34.196129 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.196061    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-workload-certs" pod "3c4543ad-ea76-4398-b7f2-7d7a32132821" (UID: "3c4543ad-ea76-4398-b7f2-7d7a32132821"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:34.196191 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.196111    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-credential-socket" pod "3c4543ad-ea76-4398-b7f2-7d7a32132821" (UID: "3c4543ad-ea76-4398-b7f2-7d7a32132821"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:34.196191 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.196180    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-workload-socket" pod "3c4543ad-ea76-4398-b7f2-7d7a32132821" (UID: "3c4543ad-ea76-4398-b7f2-7d7a32132821"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:34.196299 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.196252    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4543ad-ea76-4398-b7f2-7d7a32132821-istiod-ca-cert" pod "3c4543ad-ea76-4398-b7f2-7d7a32132821" (UID: "3c4543ad-ea76-4398-b7f2-7d7a32132821"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:34.196537 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.196517    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:34.196537 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.196536    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:34.196678 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.196547    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:34.196678 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.196556    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4543ad-ea76-4398-b7f2-7d7a32132821-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:34.196678 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.196564    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:34.198119 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.198091    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-podinfo" pod "3c4543ad-ea76-4398-b7f2-7d7a32132821" (UID: "3c4543ad-ea76-4398-b7f2-7d7a32132821"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:12:34.198323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.198121    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-token" pod "3c4543ad-ea76-4398-b7f2-7d7a32132821" (UID: "3c4543ad-ea76-4398-b7f2-7d7a32132821"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:34.198417 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.198400    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4543ad-ea76-4398-b7f2-7d7a32132821-kube-api-access-4swls" pod "3c4543ad-ea76-4398-b7f2-7d7a32132821" (UID: "3c4543ad-ea76-4398-b7f2-7d7a32132821"). InnerVolumeSpecName "kube-api-access-4swls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:34.198478 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.198436    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-envoy" pod "3c4543ad-ea76-4398-b7f2-7d7a32132821" (UID: "3c4543ad-ea76-4398-b7f2-7d7a32132821"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:34.280385 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.280350    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="144ed52c-29ff-42f9-8695-701aef451ce5" path="/var/lib/kubelet/pods/144ed52c-29ff-42f9-8695-701aef451ce5/volumes"
Oct 30 20:12:34.280787 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.280773    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e95f14-dd0c-47d5-92ef-5acc29210a5d" path="/var/lib/kubelet/pods/60e95f14-dd0c-47d5-92ef-5acc29210a5d/volumes"
Oct 30 20:12:34.297715 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.297677    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:34.297715 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.297705    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:34.297975 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.297719    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4543ad-ea76-4398-b7f2-7d7a32132821-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:34.297975 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.297751    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4swls\" (UniqueName: \"kubernetes.io/projected/3c4543ad-ea76-4398-b7f2-7d7a32132821-kube-api-access-4swls\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:34.764464 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.764432    4823 generic.go:356] "Generic (PLEG): container finished" podID="3c4543ad-ea76-4398-b7f2-7d7a32132821" containerID="d6eb2da20b5918925e4f098d2183523084dc4ec1ad8a043136ca04ede720116a" exitCode=0
Oct 30 20:12:34.764921 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.764475    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"
Oct 30 20:12:34.764921 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.764505    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw" event={"ID":"3c4543ad-ea76-4398-b7f2-7d7a32132821","Type":"ContainerDied","Data":"d6eb2da20b5918925e4f098d2183523084dc4ec1ad8a043136ca04ede720116a"}
Oct 30 20:12:34.764921 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.764558    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw" event={"ID":"3c4543ad-ea76-4398-b7f2-7d7a32132821","Type":"ContainerDied","Data":"a420ce8f09181eb46c69ecc6302d5a7eb1e663858452f864a0098c651728837e"}
Oct 30 20:12:34.764921 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.764581    4823 scope.go:122] "RemoveContainer" containerID="d6eb2da20b5918925e4f098d2183523084dc4ec1ad8a043136ca04ede720116a"
Oct 30 20:12:34.778060 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.778026    4823 scope.go:122] "RemoveContainer" containerID="d6eb2da20b5918925e4f098d2183523084dc4ec1ad8a043136ca04ede720116a"
Oct 30 20:12:34.778383 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:12:34.778354    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6eb2da20b5918925e4f098d2183523084dc4ec1ad8a043136ca04ede720116a\": container with ID starting with d6eb2da20b5918925e4f098d2183523084dc4ec1ad8a043136ca04ede720116a not found: ID does not exist" containerID="d6eb2da20b5918925e4f098d2183523084dc4ec1ad8a043136ca04ede720116a"
Oct 30 20:12:34.778478 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.778393    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6eb2da20b5918925e4f098d2183523084dc4ec1ad8a043136ca04ede720116a"} err="failed to get container status \"d6eb2da20b5918925e4f098d2183523084dc4ec1ad8a043136ca04ede720116a\": rpc error: code = NotFound desc = could not find container \"d6eb2da20b5918925e4f098d2183523084dc4ec1ad8a043136ca04ede720116a\": container with ID starting with d6eb2da20b5918925e4f098d2183523084dc4ec1ad8a043136ca04ede720116a not found: ID does not exist"
Oct 30 20:12:34.785282 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.785254    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"]
Oct 30 20:12:34.789143 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.789117    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b0d13a-6d9dl"]
Oct 30 20:12:34.816874 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.816826    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"]
Oct 30 20:12:34.824889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.824836    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-bmnvw"]
Oct 30 20:12:34.974980 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:34.974954    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"
Oct 30 20:12:35.033286 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:35.033196    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-kube-api-access-pswk7\" (UniqueName: \"kubernetes.io/projected/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-kube-api-access-pswk7\") pod \"432a6a67-fb8e-46da-aa1d-61aa0f4721a1\" (UID: \"432a6a67-fb8e-46da-aa1d-61aa0f4721a1\") "
Oct 30 20:12:35.033462 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:35.033421    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-maas-api-openshift-ca\") pod \"432a6a67-fb8e-46da-aa1d-61aa0f4721a1\" (UID: \"432a6a67-fb8e-46da-aa1d-61aa0f4721a1\") "
Oct 30 20:12:35.033531 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:35.033459    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-maas-api-xks-ca\") pod \"432a6a67-fb8e-46da-aa1d-61aa0f4721a1\" (UID: \"432a6a67-fb8e-46da-aa1d-61aa0f4721a1\") "
Oct 30 20:12:35.033949 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:35.033840    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-maas-api-openshift-ca" pod "432a6a67-fb8e-46da-aa1d-61aa0f4721a1" (UID: "432a6a67-fb8e-46da-aa1d-61aa0f4721a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:35.034178 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:35.034143    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:35.035554 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:35.035532    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-kube-api-access-pswk7" pod "432a6a67-fb8e-46da-aa1d-61aa0f4721a1" (UID: "432a6a67-fb8e-46da-aa1d-61aa0f4721a1"). InnerVolumeSpecName "kube-api-access-pswk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:35.036213 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:35.036191    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-maas-api-xks-ca" pod "432a6a67-fb8e-46da-aa1d-61aa0f4721a1" (UID: "432a6a67-fb8e-46da-aa1d-61aa0f4721a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:35.135084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:35.135053    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:35.135084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:35.135081    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pswk7\" (UniqueName: \"kubernetes.io/projected/432a6a67-fb8e-46da-aa1d-61aa0f4721a1-kube-api-access-pswk7\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:35.775109 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:35.775075    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm" event={"ID":"432a6a67-fb8e-46da-aa1d-61aa0f4721a1","Type":"ContainerDied","Data":"11fbcb2136892dfece5c8d70258b9f90295bd757ed51db3c00ac1cade9fb36aa"}
Oct 30 20:12:35.775109 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:35.775102    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"
Oct 30 20:12:35.775570 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:35.775128    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11fbcb2136892dfece5c8d70258b9f90295bd757ed51db3c00ac1cade9fb36aa"
Oct 30 20:12:36.280428 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:36.280393    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4543ad-ea76-4398-b7f2-7d7a32132821" path="/var/lib/kubelet/pods/3c4543ad-ea76-4398-b7f2-7d7a32132821/volumes"
Oct 30 20:12:36.280892 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:36.280878    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="669b522e-bca0-45a3-a6c5-b79a3460414a" path="/var/lib/kubelet/pods/669b522e-bca0-45a3-a6c5-b79a3460414a/volumes"
Oct 30 20:12:37.681937 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:37.670855    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"]
Oct 30 20:12:37.681937 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:37.671190    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn" podUID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" containerName="istio-proxy" containerID="cri-o://9368584fc56201e94bc1ddecc78d79adc0425b33e08f8ef0b2b55da7e7afed66" gracePeriod=30
Oct 30 20:12:38.406312 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.406277    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"]
Oct 30 20:12:38.406915 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.406897    4823 cpu_manager.go:414] "containerMap: removing container" podUID="3c4543ad-ea76-4398-b7f2-7d7a32132821" containerName="istio-proxy"
Oct 30 20:12:38.406971 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.406917    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4543ad-ea76-4398-b7f2-7d7a32132821" containerName="istio-proxy"
Oct 30 20:12:38.406971 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.406946    4823 cpu_manager.go:414] "containerMap: removing container" podUID="4e8bbafa-1167-49d4-a51b-a90801d5d66b" containerName="revoke-keys"
Oct 30 20:12:38.406971 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.406951    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8bbafa-1167-49d4-a51b-a90801d5d66b" containerName="revoke-keys"
Oct 30 20:12:38.406971 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.406970    4823 cpu_manager.go:414] "containerMap: removing container" podUID="432a6a67-fb8e-46da-aa1d-61aa0f4721a1" containerName="revoke-keys"
Oct 30 20:12:38.407112 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.406976    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432a6a67-fb8e-46da-aa1d-61aa0f4721a1" containerName="revoke-keys"
Oct 30 20:12:38.407112 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.407096    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8bbafa-1167-49d4-a51b-a90801d5d66b" containerName="revoke-keys"
Oct 30 20:12:38.407112 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.407108    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="432a6a67-fb8e-46da-aa1d-61aa0f4721a1" containerName="revoke-keys"
Oct 30 20:12:38.407208 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.407120    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4543ad-ea76-4398-b7f2-7d7a32132821" containerName="istio-proxy"
Oct 30 20:12:38.411926 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.411905    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"
Oct 30 20:12:38.420679 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.420652    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:12:38.421382 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.421355    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:12:38.421848 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.421820    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:12:38.422135 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.422116    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:12:38.424684 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.424662    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"]
Oct 30 20:12:38.495749 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.495712    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b5959c-301e-4849-91c3-f8ffecd21731-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4\" (UID: \"71b5959c-301e-4849-91c3-f8ffecd21731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"
Oct 30 20:12:38.495949 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.495929    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpf4z\" (UniqueName: \"kubernetes.io/projected/71b5959c-301e-4849-91c3-f8ffecd21731-kube-api-access-bpf4z\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4\" (UID: \"71b5959c-301e-4849-91c3-f8ffecd21731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"
Oct 30 20:12:38.496108 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.496091    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b5959c-301e-4849-91c3-f8ffecd21731-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4\" (UID: \"71b5959c-301e-4849-91c3-f8ffecd21731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"
Oct 30 20:12:38.597585 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.597553    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b5959c-301e-4849-91c3-f8ffecd21731-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4\" (UID: \"71b5959c-301e-4849-91c3-f8ffecd21731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"
Oct 30 20:12:38.597585 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.597593    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b5959c-301e-4849-91c3-f8ffecd21731-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4\" (UID: \"71b5959c-301e-4849-91c3-f8ffecd21731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"
Oct 30 20:12:38.597848 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.597682    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpf4z\" (UniqueName: \"kubernetes.io/projected/71b5959c-301e-4849-91c3-f8ffecd21731-kube-api-access-bpf4z\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4\" (UID: \"71b5959c-301e-4849-91c3-f8ffecd21731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"
Oct 30 20:12:38.598309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.598283    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b5959c-301e-4849-91c3-f8ffecd21731-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4\" (UID: \"71b5959c-301e-4849-91c3-f8ffecd21731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"
Oct 30 20:12:38.600120 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.600090    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b5959c-301e-4849-91c3-f8ffecd21731-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4\" (UID: \"71b5959c-301e-4849-91c3-f8ffecd21731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"
Oct 30 20:12:38.612468 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.612443    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpf4z\" (UniqueName: \"kubernetes.io/projected/71b5959c-301e-4849-91c3-f8ffecd21731-kube-api-access-bpf4z\") pod \"maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4\" (UID: \"71b5959c-301e-4849-91c3-f8ffecd21731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"
Oct 30 20:12:38.730645 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:38.730571    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"
Oct 30 20:12:39.045590 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:39.045013    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"]
Oct 30 20:12:39.052649 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:12:39.052618    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b5959c_301e_4849_91c3_f8ffecd21731.slice/crio-f0f0ceb6d9993c70dd4404361d0f3726c2ee842a7cf3d7a29828d615052cce21 WatchSource:0}: Error finding container f0f0ceb6d9993c70dd4404361d0f3726c2ee842a7cf3d7a29828d615052cce21: Status 404 returned error can't find the container with id f0f0ceb6d9993c70dd4404361d0f3726c2ee842a7cf3d7a29828d615052cce21
Oct 30 20:12:39.859352 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:39.859316    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4" event={"ID":"71b5959c-301e-4849-91c3-f8ffecd21731","Type":"ContainerStarted","Data":"21bf52077725f14644124cef293c84926529b0494e85fa858632ddd68cff0de9"}
Oct 30 20:12:39.859749 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:39.859375    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4" event={"ID":"71b5959c-301e-4849-91c3-f8ffecd21731","Type":"ContainerStarted","Data":"f0f0ceb6d9993c70dd4404361d0f3726c2ee842a7cf3d7a29828d615052cce21"}
Oct 30 20:12:39.894810 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:39.894753    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4" podStartSLOduration=1.8947386320000001 podStartE2EDuration="1.894738632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:39.888087308 +0000 UTC m=+2559.780701634" watchObservedRunningTime="2026-10-30 20:12:39.894738632 +0000 UTC m=+2559.787352987"
Oct 30 20:12:40.869731 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:40.869694    4823 generic.go:356] "Generic (PLEG): container finished" podID="71b5959c-301e-4849-91c3-f8ffecd21731" containerID="21bf52077725f14644124cef293c84926529b0494e85fa858632ddd68cff0de9" exitCode=0
Oct 30 20:12:40.870160 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:40.869777    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4" event={"ID":"71b5959c-301e-4849-91c3-f8ffecd21731","Type":"ContainerDied","Data":"21bf52077725f14644124cef293c84926529b0494e85fa858632ddd68cff0de9"}
Oct 30 20:12:42.059202 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.059173    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"
Oct 30 20:12:42.155093 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.155004    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71b5959c-301e-4849-91c3-f8ffecd21731-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b5959c-301e-4849-91c3-f8ffecd21731-maas-api-xks-ca\") pod \"71b5959c-301e-4849-91c3-f8ffecd21731\" (UID: \"71b5959c-301e-4849-91c3-f8ffecd21731\") "
Oct 30 20:12:42.155093 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.155047    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b5959c-301e-4849-91c3-f8ffecd21731-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b5959c-301e-4849-91c3-f8ffecd21731-maas-api-openshift-ca\") pod \"71b5959c-301e-4849-91c3-f8ffecd21731\" (UID: \"71b5959c-301e-4849-91c3-f8ffecd21731\") "
Oct 30 20:12:42.155340 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.155172    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b5959c-301e-4849-91c3-f8ffecd21731-kube-api-access-bpf4z\" (UniqueName: \"kubernetes.io/projected/71b5959c-301e-4849-91c3-f8ffecd21731-kube-api-access-bpf4z\") pod \"71b5959c-301e-4849-91c3-f8ffecd21731\" (UID: \"71b5959c-301e-4849-91c3-f8ffecd21731\") "
Oct 30 20:12:42.155462 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.155435    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b5959c-301e-4849-91c3-f8ffecd21731-maas-api-openshift-ca" pod "71b5959c-301e-4849-91c3-f8ffecd21731" (UID: "71b5959c-301e-4849-91c3-f8ffecd21731"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:42.155802 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.155782    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b5959c-301e-4849-91c3-f8ffecd21731-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:42.157311 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.157288    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b5959c-301e-4849-91c3-f8ffecd21731-maas-api-xks-ca" pod "71b5959c-301e-4849-91c3-f8ffecd21731" (UID: "71b5959c-301e-4849-91c3-f8ffecd21731"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:42.157550 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.157526    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b5959c-301e-4849-91c3-f8ffecd21731-kube-api-access-bpf4z" pod "71b5959c-301e-4849-91c3-f8ffecd21731" (UID: "71b5959c-301e-4849-91c3-f8ffecd21731"). InnerVolumeSpecName "kube-api-access-bpf4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:42.166844 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.166815    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:12:42.166966 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.166899    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn" podUID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:12:42.257352 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.257318    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpf4z\" (UniqueName: \"kubernetes.io/projected/71b5959c-301e-4849-91c3-f8ffecd21731-kube-api-access-bpf4z\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:42.257352 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.257350    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b5959c-301e-4849-91c3-f8ffecd21731-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:42.901429 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.900978    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"
Oct 30 20:12:42.901429 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.900974    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4" event={"ID":"71b5959c-301e-4849-91c3-f8ffecd21731","Type":"ContainerDied","Data":"f0f0ceb6d9993c70dd4404361d0f3726c2ee842a7cf3d7a29828d615052cce21"}
Oct 30 20:12:42.901429 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.901145    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0f0ceb6d9993c70dd4404361d0f3726c2ee842a7cf3d7a29828d615052cce21"
Oct 30 20:12:42.909661 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.904332    4823 generic.go:356] "Generic (PLEG): container finished" podID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" containerID="9368584fc56201e94bc1ddecc78d79adc0425b33e08f8ef0b2b55da7e7afed66" exitCode=0
Oct 30 20:12:42.909661 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.904425    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn" event={"ID":"cfe9f1d8-aef1-41cd-b586-c0c073d9b574","Type":"ContainerDied","Data":"9368584fc56201e94bc1ddecc78d79adc0425b33e08f8ef0b2b55da7e7afed66"}
Oct 30 20:12:42.909661 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.904504    4823 scope.go:122] "RemoveContainer" containerID="6d7549911361640349a52a5388f57952e449b8f0a495191422cd0c13a9c248d1"
Oct 30 20:12:42.986971 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:42.986946    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 20:12:43.170167 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.170125    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istiod-ca-cert\") pod \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") "
Oct 30 20:12:43.170568 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.170204    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-workload-socket\") pod \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") "
Oct 30 20:12:43.170568 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.170283    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-envoy\") pod \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") "
Oct 30 20:12:43.170568 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.170322    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-credential-socket\") pod \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") "
Oct 30 20:12:43.170568 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.170344    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-podinfo\") pod \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") "
Oct 30 20:12:43.170568 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.170416    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-workload-certs\") pod \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") "
Oct 30 20:12:43.170568 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.170495    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-token\" (UniqueName: \"kubernetes.io/projected/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-token\") pod \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") "
Oct 30 20:12:43.170568 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.170524    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-kube-api-access-kdrmr\" (UniqueName: \"kubernetes.io/projected/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-kube-api-access-kdrmr\") pod \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") "
Oct 30 20:12:43.170568 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.170561    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-data\") pod \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\" (UID: \"cfe9f1d8-aef1-41cd-b586-c0c073d9b574\") "
Oct 30 20:12:43.171027 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.170566    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-workload-socket" pod "cfe9f1d8-aef1-41cd-b586-c0c073d9b574" (UID: "cfe9f1d8-aef1-41cd-b586-c0c073d9b574"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:43.171027 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.170619    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-credential-socket" pod "cfe9f1d8-aef1-41cd-b586-c0c073d9b574" (UID: "cfe9f1d8-aef1-41cd-b586-c0c073d9b574"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:43.171027 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.170628    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istiod-ca-cert" pod "cfe9f1d8-aef1-41cd-b586-c0c073d9b574" (UID: "cfe9f1d8-aef1-41cd-b586-c0c073d9b574"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:43.171027 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.170830    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-workload-certs" pod "cfe9f1d8-aef1-41cd-b586-c0c073d9b574" (UID: "cfe9f1d8-aef1-41cd-b586-c0c073d9b574"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:43.171027 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.170924    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-data" pod "cfe9f1d8-aef1-41cd-b586-c0c073d9b574" (UID: "cfe9f1d8-aef1-41cd-b586-c0c073d9b574"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:43.171313 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.171290    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:43.171374 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.171322    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:43.171374 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.171338    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:43.171374 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.171353    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:43.171374 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.171367    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:43.172753 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.172729    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-podinfo" pod "cfe9f1d8-aef1-41cd-b586-c0c073d9b574" (UID: "cfe9f1d8-aef1-41cd-b586-c0c073d9b574"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:12:43.172810 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.172730    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-kube-api-access-kdrmr" pod "cfe9f1d8-aef1-41cd-b586-c0c073d9b574" (UID: "cfe9f1d8-aef1-41cd-b586-c0c073d9b574"). InnerVolumeSpecName "kube-api-access-kdrmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:43.173085 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.173055    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-envoy" pod "cfe9f1d8-aef1-41cd-b586-c0c073d9b574" (UID: "cfe9f1d8-aef1-41cd-b586-c0c073d9b574"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:43.173205 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.173090    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-token" pod "cfe9f1d8-aef1-41cd-b586-c0c073d9b574" (UID: "cfe9f1d8-aef1-41cd-b586-c0c073d9b574"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:43.227266 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.227196    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"]
Oct 30 20:12:43.273137 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.273105    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:43.273137 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.273133    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdrmr\" (UniqueName: \"kubernetes.io/projected/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-kube-api-access-kdrmr\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:43.273339 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.273146    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:43.273339 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.273161    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe9f1d8-aef1-41cd-b586-c0c073d9b574-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:43.914411 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.914383    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"
Oct 30 20:12:43.914584 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.914381    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn" event={"ID":"cfe9f1d8-aef1-41cd-b586-c0c073d9b574","Type":"ContainerDied","Data":"a58c914dea42cc2fdca063c06ae17957ac61389a7f45c663d09d189260174d5c"}
Oct 30 20:12:43.914584 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.914522    4823 scope.go:122] "RemoveContainer" containerID="9368584fc56201e94bc1ddecc78d79adc0425b33e08f8ef0b2b55da7e7afed66"
Oct 30 20:12:43.940050 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.940019    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"]
Oct 30 20:12:43.946378 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:43.946353    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f5d176-openshift-default-6847b887b7-csptn"]
Oct 30 20:12:44.279913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.279825    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" path="/var/lib/kubelet/pods/cfe9f1d8-aef1-41cd-b586-c0c073d9b574/volumes"
Oct 30 20:12:44.505463 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.505432    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"]
Oct 30 20:12:44.506423 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.506392    4823 cpu_manager.go:414] "containerMap: removing container" podUID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" containerName="istio-proxy"
Oct 30 20:12:44.506423 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.506423    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" containerName="istio-proxy"
Oct 30 20:12:44.506609 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.506451    4823 cpu_manager.go:414] "containerMap: removing container" podUID="71b5959c-301e-4849-91c3-f8ffecd21731" containerName="revoke-keys"
Oct 30 20:12:44.506609 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.506461    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b5959c-301e-4849-91c3-f8ffecd21731" containerName="revoke-keys"
Oct 30 20:12:44.506609 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.506495    4823 cpu_manager.go:414] "containerMap: removing container" podUID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" containerName="istio-proxy"
Oct 30 20:12:44.506609 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.506504    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" containerName="istio-proxy"
Oct 30 20:12:44.506735 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.506657    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" containerName="istio-proxy"
Oct 30 20:12:44.506735 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.506671    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b5959c-301e-4849-91c3-f8ffecd21731" containerName="revoke-keys"
Oct 30 20:12:44.506735 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.506685    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe9f1d8-aef1-41cd-b586-c0c073d9b574" containerName="istio-proxy"
Oct 30 20:12:44.512143 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.512124    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"
Oct 30 20:12:44.518198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.518170    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"]
Oct 30 20:12:44.521204 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.521178    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:12:44.521372 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.521257    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:12:44.521372 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.521286    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:12:44.521751 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.521730    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:12:44.522603 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.522581    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"
Oct 30 20:12:44.525916 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.525896    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"]
Oct 30 20:12:44.533043 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.533022    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"]
Oct 30 20:12:44.599041 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.599004    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4sn5\" (UniqueName: \"kubernetes.io/projected/697e7d67-0ada-4ce2-968d-fb52077793bc-kube-api-access-n4sn5\") pod \"maas-api-revoke-sub-780b2274-swx9p\" (UID: \"697e7d67-0ada-4ce2-968d-fb52077793bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"
Oct 30 20:12:44.599193 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.599129    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c721c44-53fa-40b8-8684-785a1cc884f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaac20b8-5645z\" (UID: \"0c721c44-53fa-40b8-8684-785a1cc884f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"
Oct 30 20:12:44.599257 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.599240    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/697e7d67-0ada-4ce2-968d-fb52077793bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-780b2274-swx9p\" (UID: \"697e7d67-0ada-4ce2-968d-fb52077793bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"
Oct 30 20:12:44.599317 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.599277    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/697e7d67-0ada-4ce2-968d-fb52077793bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-780b2274-swx9p\" (UID: \"697e7d67-0ada-4ce2-968d-fb52077793bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"
Oct 30 20:12:44.599351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.599323    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgq5z\" (UniqueName: \"kubernetes.io/projected/0c721c44-53fa-40b8-8684-785a1cc884f3-kube-api-access-kgq5z\") pod \"maas-api-revoke-sub-aaac20b8-5645z\" (UID: \"0c721c44-53fa-40b8-8684-785a1cc884f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"
Oct 30 20:12:44.599493 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.599473    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c721c44-53fa-40b8-8684-785a1cc884f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaac20b8-5645z\" (UID: \"0c721c44-53fa-40b8-8684-785a1cc884f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"
Oct 30 20:12:44.700462 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.700428    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c721c44-53fa-40b8-8684-785a1cc884f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaac20b8-5645z\" (UID: \"0c721c44-53fa-40b8-8684-785a1cc884f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"
Oct 30 20:12:44.700649 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.700523    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4sn5\" (UniqueName: \"kubernetes.io/projected/697e7d67-0ada-4ce2-968d-fb52077793bc-kube-api-access-n4sn5\") pod \"maas-api-revoke-sub-780b2274-swx9p\" (UID: \"697e7d67-0ada-4ce2-968d-fb52077793bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"
Oct 30 20:12:44.700649 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.700569    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c721c44-53fa-40b8-8684-785a1cc884f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaac20b8-5645z\" (UID: \"0c721c44-53fa-40b8-8684-785a1cc884f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"
Oct 30 20:12:44.700765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.700662    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/697e7d67-0ada-4ce2-968d-fb52077793bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-780b2274-swx9p\" (UID: \"697e7d67-0ada-4ce2-968d-fb52077793bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"
Oct 30 20:12:44.700765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.700701    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/697e7d67-0ada-4ce2-968d-fb52077793bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-780b2274-swx9p\" (UID: \"697e7d67-0ada-4ce2-968d-fb52077793bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"
Oct 30 20:12:44.700765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.700746    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgq5z\" (UniqueName: \"kubernetes.io/projected/0c721c44-53fa-40b8-8684-785a1cc884f3-kube-api-access-kgq5z\") pod \"maas-api-revoke-sub-aaac20b8-5645z\" (UID: \"0c721c44-53fa-40b8-8684-785a1cc884f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"
Oct 30 20:12:44.701305 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.701270    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c721c44-53fa-40b8-8684-785a1cc884f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaac20b8-5645z\" (UID: \"0c721c44-53fa-40b8-8684-785a1cc884f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"
Oct 30 20:12:44.701527 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.701502    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/697e7d67-0ada-4ce2-968d-fb52077793bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-780b2274-swx9p\" (UID: \"697e7d67-0ada-4ce2-968d-fb52077793bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"
Oct 30 20:12:44.703295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.703269    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c721c44-53fa-40b8-8684-785a1cc884f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaac20b8-5645z\" (UID: \"0c721c44-53fa-40b8-8684-785a1cc884f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"
Oct 30 20:12:44.703374 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.703269    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/697e7d67-0ada-4ce2-968d-fb52077793bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-780b2274-swx9p\" (UID: \"697e7d67-0ada-4ce2-968d-fb52077793bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"
Oct 30 20:12:44.712343 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.712320    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgq5z\" (UniqueName: \"kubernetes.io/projected/0c721c44-53fa-40b8-8684-785a1cc884f3-kube-api-access-kgq5z\") pod \"maas-api-revoke-sub-aaac20b8-5645z\" (UID: \"0c721c44-53fa-40b8-8684-785a1cc884f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"
Oct 30 20:12:44.712443 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.712393    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4sn5\" (UniqueName: \"kubernetes.io/projected/697e7d67-0ada-4ce2-968d-fb52077793bc-kube-api-access-n4sn5\") pod \"maas-api-revoke-sub-780b2274-swx9p\" (UID: \"697e7d67-0ada-4ce2-968d-fb52077793bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"
Oct 30 20:12:44.828217 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.828144    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"
Oct 30 20:12:44.846201 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:44.846161    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"
Oct 30 20:12:45.152241 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:45.152211    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"]
Oct 30 20:12:45.153354 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:12:45.153329    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697e7d67_0ada_4ce2_968d_fb52077793bc.slice/crio-4c30969bdb134da42a770bc16bc7eeeafbcbcc2ce0f7dd5055e1009ffd92e874 WatchSource:0}: Error finding container 4c30969bdb134da42a770bc16bc7eeeafbcbcc2ce0f7dd5055e1009ffd92e874: Status 404 returned error can't find the container with id 4c30969bdb134da42a770bc16bc7eeeafbcbcc2ce0f7dd5055e1009ffd92e874
Oct 30 20:12:45.169877 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:45.169834    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"]
Oct 30 20:12:45.170534 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:12:45.170508    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c721c44_53fa_40b8_8684_785a1cc884f3.slice/crio-ef4954c4bcd0accc32e44ec0c15652dcbf6abbc0b89ab8e19ef960a44b78bb2d WatchSource:0}: Error finding container ef4954c4bcd0accc32e44ec0c15652dcbf6abbc0b89ab8e19ef960a44b78bb2d: Status 404 returned error can't find the container with id ef4954c4bcd0accc32e44ec0c15652dcbf6abbc0b89ab8e19ef960a44b78bb2d
Oct 30 20:12:45.944638 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:45.944597    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z" event={"ID":"0c721c44-53fa-40b8-8684-785a1cc884f3","Type":"ContainerStarted","Data":"da6449a24ab460eab84af1f4d3bf9aa737a7b7e80b4f225beb7aa007ac49d4f2"}
Oct 30 20:12:45.945102 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:45.944687    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z" event={"ID":"0c721c44-53fa-40b8-8684-785a1cc884f3","Type":"ContainerStarted","Data":"ef4954c4bcd0accc32e44ec0c15652dcbf6abbc0b89ab8e19ef960a44b78bb2d"}
Oct 30 20:12:45.946599 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:45.946571    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p" event={"ID":"697e7d67-0ada-4ce2-968d-fb52077793bc","Type":"ContainerStarted","Data":"f5ed87df1903902e1f958108bcfecf7e40bbd502d156a2d0b51d4b80b461cb50"}
Oct 30 20:12:45.946719 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:45.946615    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p" event={"ID":"697e7d67-0ada-4ce2-968d-fb52077793bc","Type":"ContainerStarted","Data":"4c30969bdb134da42a770bc16bc7eeeafbcbcc2ce0f7dd5055e1009ffd92e874"}
Oct 30 20:12:45.977242 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:45.977192    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z" podStartSLOduration=1.977179309 podStartE2EDuration="1.977179309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:45.970797664 +0000 UTC m=+2565.863411988" watchObservedRunningTime="2026-10-30 20:12:45.977179309 +0000 UTC m=+2565.869793634"
Oct 30 20:12:46.012250 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:46.012200    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p" podStartSLOduration=2.01218619 podStartE2EDuration="2.01218619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:46.009130231 +0000 UTC m=+2565.901744556" watchObservedRunningTime="2026-10-30 20:12:46.01218619 +0000 UTC m=+2565.904800559"
Oct 30 20:12:46.956202 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:46.956166    4823 generic.go:356] "Generic (PLEG): container finished" podID="0c721c44-53fa-40b8-8684-785a1cc884f3" containerID="da6449a24ab460eab84af1f4d3bf9aa737a7b7e80b4f225beb7aa007ac49d4f2" exitCode=0
Oct 30 20:12:46.956602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:46.956257    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z" event={"ID":"0c721c44-53fa-40b8-8684-785a1cc884f3","Type":"ContainerDied","Data":"da6449a24ab460eab84af1f4d3bf9aa737a7b7e80b4f225beb7aa007ac49d4f2"}
Oct 30 20:12:46.958059 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:46.958032    4823 generic.go:356] "Generic (PLEG): container finished" podID="697e7d67-0ada-4ce2-968d-fb52077793bc" containerID="f5ed87df1903902e1f958108bcfecf7e40bbd502d156a2d0b51d4b80b461cb50" exitCode=0
Oct 30 20:12:46.958210 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:46.958110    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p" event={"ID":"697e7d67-0ada-4ce2-968d-fb52077793bc","Type":"ContainerDied","Data":"f5ed87df1903902e1f958108bcfecf7e40bbd502d156a2d0b51d4b80b461cb50"}
Oct 30 20:12:48.167972 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.167932    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"
Oct 30 20:12:48.201924 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.201899    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"
Oct 30 20:12:48.250753 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.250658    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/697e7d67-0ada-4ce2-968d-fb52077793bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/697e7d67-0ada-4ce2-968d-fb52077793bc-maas-api-openshift-ca\") pod \"697e7d67-0ada-4ce2-968d-fb52077793bc\" (UID: \"697e7d67-0ada-4ce2-968d-fb52077793bc\") "
Oct 30 20:12:48.250753 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.250710    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c721c44-53fa-40b8-8684-785a1cc884f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c721c44-53fa-40b8-8684-785a1cc884f3-maas-api-xks-ca\") pod \"0c721c44-53fa-40b8-8684-785a1cc884f3\" (UID: \"0c721c44-53fa-40b8-8684-785a1cc884f3\") "
Oct 30 20:12:48.251015 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.250810    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/697e7d67-0ada-4ce2-968d-fb52077793bc-kube-api-access-n4sn5\" (UniqueName: \"kubernetes.io/projected/697e7d67-0ada-4ce2-968d-fb52077793bc-kube-api-access-n4sn5\") pod \"697e7d67-0ada-4ce2-968d-fb52077793bc\" (UID: \"697e7d67-0ada-4ce2-968d-fb52077793bc\") "
Oct 30 20:12:48.251015 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.250931    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/697e7d67-0ada-4ce2-968d-fb52077793bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/697e7d67-0ada-4ce2-968d-fb52077793bc-maas-api-xks-ca\") pod \"697e7d67-0ada-4ce2-968d-fb52077793bc\" (UID: \"697e7d67-0ada-4ce2-968d-fb52077793bc\") "
Oct 30 20:12:48.251198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.251078    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c721c44-53fa-40b8-8684-785a1cc884f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c721c44-53fa-40b8-8684-785a1cc884f3-maas-api-openshift-ca\") pod \"0c721c44-53fa-40b8-8684-785a1cc884f3\" (UID: \"0c721c44-53fa-40b8-8684-785a1cc884f3\") "
Oct 30 20:12:48.251198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.251080    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/697e7d67-0ada-4ce2-968d-fb52077793bc-maas-api-openshift-ca" pod "697e7d67-0ada-4ce2-968d-fb52077793bc" (UID: "697e7d67-0ada-4ce2-968d-fb52077793bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:48.251198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.251190    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c721c44-53fa-40b8-8684-785a1cc884f3-kube-api-access-kgq5z\" (UniqueName: \"kubernetes.io/projected/0c721c44-53fa-40b8-8684-785a1cc884f3-kube-api-access-kgq5z\") pod \"0c721c44-53fa-40b8-8684-785a1cc884f3\" (UID: \"0c721c44-53fa-40b8-8684-785a1cc884f3\") "
Oct 30 20:12:48.251566 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.251535    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c721c44-53fa-40b8-8684-785a1cc884f3-maas-api-openshift-ca" pod "0c721c44-53fa-40b8-8684-785a1cc884f3" (UID: "0c721c44-53fa-40b8-8684-785a1cc884f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:48.252013 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.251989    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c721c44-53fa-40b8-8684-785a1cc884f3-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:48.252145 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.252018    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/697e7d67-0ada-4ce2-968d-fb52077793bc-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:48.258403 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.258371    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c721c44-53fa-40b8-8684-785a1cc884f3-maas-api-xks-ca" pod "0c721c44-53fa-40b8-8684-785a1cc884f3" (UID: "0c721c44-53fa-40b8-8684-785a1cc884f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:48.258491 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.258406    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c721c44-53fa-40b8-8684-785a1cc884f3-kube-api-access-kgq5z" pod "0c721c44-53fa-40b8-8684-785a1cc884f3" (UID: "0c721c44-53fa-40b8-8684-785a1cc884f3"). InnerVolumeSpecName "kube-api-access-kgq5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:48.258491 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.258441    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697e7d67-0ada-4ce2-968d-fb52077793bc-kube-api-access-n4sn5" pod "697e7d67-0ada-4ce2-968d-fb52077793bc" (UID: "697e7d67-0ada-4ce2-968d-fb52077793bc"). InnerVolumeSpecName "kube-api-access-n4sn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:48.258491 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.258479    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/697e7d67-0ada-4ce2-968d-fb52077793bc-maas-api-xks-ca" pod "697e7d67-0ada-4ce2-968d-fb52077793bc" (UID: "697e7d67-0ada-4ce2-968d-fb52077793bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:48.353366 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.353326    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4sn5\" (UniqueName: \"kubernetes.io/projected/697e7d67-0ada-4ce2-968d-fb52077793bc-kube-api-access-n4sn5\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:48.353366 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.353359    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/697e7d67-0ada-4ce2-968d-fb52077793bc-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:48.353590 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.353373    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgq5z\" (UniqueName: \"kubernetes.io/projected/0c721c44-53fa-40b8-8684-785a1cc884f3-kube-api-access-kgq5z\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:48.353590 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.353385    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c721c44-53fa-40b8-8684-785a1cc884f3-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:48.449913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.449875    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"]
Oct 30 20:12:48.469774 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.469740    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"]
Oct 30 20:12:48.995631 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.995600    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"
Oct 30 20:12:48.995794 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.995599    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p" event={"ID":"697e7d67-0ada-4ce2-968d-fb52077793bc","Type":"ContainerDied","Data":"4c30969bdb134da42a770bc16bc7eeeafbcbcc2ce0f7dd5055e1009ffd92e874"}
Oct 30 20:12:48.995794 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.995774    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c30969bdb134da42a770bc16bc7eeeafbcbcc2ce0f7dd5055e1009ffd92e874"
Oct 30 20:12:48.997172 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.997152    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"
Oct 30 20:12:48.997172 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.997155    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z" event={"ID":"0c721c44-53fa-40b8-8684-785a1cc884f3","Type":"ContainerDied","Data":"ef4954c4bcd0accc32e44ec0c15652dcbf6abbc0b89ab8e19ef960a44b78bb2d"}
Oct 30 20:12:48.997321 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:48.997199    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef4954c4bcd0accc32e44ec0c15652dcbf6abbc0b89ab8e19ef960a44b78bb2d"
Oct 30 20:12:52.698686 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.698656    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"]
Oct 30 20:12:52.699298 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.699281    4823 cpu_manager.go:414] "containerMap: removing container" podUID="697e7d67-0ada-4ce2-968d-fb52077793bc" containerName="revoke-keys"
Oct 30 20:12:52.699346 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.699300    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="697e7d67-0ada-4ce2-968d-fb52077793bc" containerName="revoke-keys"
Oct 30 20:12:52.699346 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.699330    4823 cpu_manager.go:414] "containerMap: removing container" podUID="0c721c44-53fa-40b8-8684-785a1cc884f3" containerName="revoke-keys"
Oct 30 20:12:52.699346 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.699337    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c721c44-53fa-40b8-8684-785a1cc884f3" containerName="revoke-keys"
Oct 30 20:12:52.699464 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.699453    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="697e7d67-0ada-4ce2-968d-fb52077793bc" containerName="revoke-keys"
Oct 30 20:12:52.699501 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.699469    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c721c44-53fa-40b8-8684-785a1cc884f3" containerName="revoke-keys"
Oct 30 20:12:52.704040 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.704012    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"
Oct 30 20:12:52.711643 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.711613    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"]
Oct 30 20:12:52.724141 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.716331    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"
Oct 30 20:12:52.735682 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.735661    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:12:52.736050 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.736032    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:12:52.736253 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.736239    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:12:52.737160 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.737141    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:12:52.741346 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.741325    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"]
Oct 30 20:12:52.747170 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.747147    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"]
Oct 30 20:12:52.857366 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.857330    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b6589b-3575-4241-820e-d17634751dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fa7f545-kfld5\" (UID: \"c6b6589b-3575-4241-820e-d17634751dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"
Oct 30 20:12:52.857366 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.857365    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzrf7\" (UniqueName: \"kubernetes.io/projected/c6b6589b-3575-4241-820e-d17634751dac-kube-api-access-nzrf7\") pod \"maas-api-revoke-sub-4fa7f545-kfld5\" (UID: \"c6b6589b-3575-4241-820e-d17634751dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"
Oct 30 20:12:52.857629 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.857605    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7081733e-6a21-4108-bd76-40c7e4d98ea6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f14ab15-9zjfz\" (UID: \"7081733e-6a21-4108-bd76-40c7e4d98ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"
Oct 30 20:12:52.857678 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.857660    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b6589b-3575-4241-820e-d17634751dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fa7f545-kfld5\" (UID: \"c6b6589b-3575-4241-820e-d17634751dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"
Oct 30 20:12:52.857718 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.857708    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7081733e-6a21-4108-bd76-40c7e4d98ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f14ab15-9zjfz\" (UID: \"7081733e-6a21-4108-bd76-40c7e4d98ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"
Oct 30 20:12:52.857775 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.857764    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4jtd\" (UniqueName: \"kubernetes.io/projected/7081733e-6a21-4108-bd76-40c7e4d98ea6-kube-api-access-p4jtd\") pod \"maas-api-revoke-sub-1f14ab15-9zjfz\" (UID: \"7081733e-6a21-4108-bd76-40c7e4d98ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"
Oct 30 20:12:52.979945 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.979705    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b6589b-3575-4241-820e-d17634751dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fa7f545-kfld5\" (UID: \"c6b6589b-3575-4241-820e-d17634751dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"
Oct 30 20:12:52.979945 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.979784    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7081733e-6a21-4108-bd76-40c7e4d98ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f14ab15-9zjfz\" (UID: \"7081733e-6a21-4108-bd76-40c7e4d98ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"
Oct 30 20:12:52.979945 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.979852    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4jtd\" (UniqueName: \"kubernetes.io/projected/7081733e-6a21-4108-bd76-40c7e4d98ea6-kube-api-access-p4jtd\") pod \"maas-api-revoke-sub-1f14ab15-9zjfz\" (UID: \"7081733e-6a21-4108-bd76-40c7e4d98ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"
Oct 30 20:12:52.979945 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.979909    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b6589b-3575-4241-820e-d17634751dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fa7f545-kfld5\" (UID: \"c6b6589b-3575-4241-820e-d17634751dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"
Oct 30 20:12:52.979945 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.979933    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzrf7\" (UniqueName: \"kubernetes.io/projected/c6b6589b-3575-4241-820e-d17634751dac-kube-api-access-nzrf7\") pod \"maas-api-revoke-sub-4fa7f545-kfld5\" (UID: \"c6b6589b-3575-4241-820e-d17634751dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"
Oct 30 20:12:52.980390 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.980210    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7081733e-6a21-4108-bd76-40c7e4d98ea6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f14ab15-9zjfz\" (UID: \"7081733e-6a21-4108-bd76-40c7e4d98ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"
Oct 30 20:12:52.986942 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.986910    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7081733e-6a21-4108-bd76-40c7e4d98ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f14ab15-9zjfz\" (UID: \"7081733e-6a21-4108-bd76-40c7e4d98ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"
Oct 30 20:12:52.998973 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:52.998937    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b6589b-3575-4241-820e-d17634751dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fa7f545-kfld5\" (UID: \"c6b6589b-3575-4241-820e-d17634751dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"
Oct 30 20:12:53.028692 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:53.001540    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b6589b-3575-4241-820e-d17634751dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fa7f545-kfld5\" (UID: \"c6b6589b-3575-4241-820e-d17634751dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"
Oct 30 20:12:53.028956 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:53.004321    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7081733e-6a21-4108-bd76-40c7e4d98ea6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f14ab15-9zjfz\" (UID: \"7081733e-6a21-4108-bd76-40c7e4d98ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"
Oct 30 20:12:53.029142 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:53.012104    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4jtd\" (UniqueName: \"kubernetes.io/projected/7081733e-6a21-4108-bd76-40c7e4d98ea6-kube-api-access-p4jtd\") pod \"maas-api-revoke-sub-1f14ab15-9zjfz\" (UID: \"7081733e-6a21-4108-bd76-40c7e4d98ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"
Oct 30 20:12:53.029289 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:53.028608    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzrf7\" (UniqueName: \"kubernetes.io/projected/c6b6589b-3575-4241-820e-d17634751dac-kube-api-access-nzrf7\") pod \"maas-api-revoke-sub-4fa7f545-kfld5\" (UID: \"c6b6589b-3575-4241-820e-d17634751dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"
Oct 30 20:12:53.053146 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:53.053107    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"
Oct 30 20:12:53.056416 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:53.056368    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"
Oct 30 20:12:53.212121 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:53.212077    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"]
Oct 30 20:12:53.218212 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:53.218185    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3f57c5-jbwj8"]
Oct 30 20:12:53.409523 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:53.409430    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"]
Oct 30 20:12:53.412973 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:12:53.412938    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7081733e_6a21_4108_bd76_40c7e4d98ea6.slice/crio-6ebad7bb3b854238f8b730fa09bc9c40f17d41a3acbd48cef0f9c7483a8f0c9d WatchSource:0}: Error finding container 6ebad7bb3b854238f8b730fa09bc9c40f17d41a3acbd48cef0f9c7483a8f0c9d: Status 404 returned error can't find the container with id 6ebad7bb3b854238f8b730fa09bc9c40f17d41a3acbd48cef0f9c7483a8f0c9d
Oct 30 20:12:53.413589 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:53.413561    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"]
Oct 30 20:12:53.420740 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:12:53.420699    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b6589b_3575_4241_820e_d17634751dac.slice/crio-b906f78d6f91ceec62f52940fc2eb5ff89e28abf79f4730fced370fa50bb2af1 WatchSource:0}: Error finding container b906f78d6f91ceec62f52940fc2eb5ff89e28abf79f4730fced370fa50bb2af1: Status 404 returned error can't find the container with id b906f78d6f91ceec62f52940fc2eb5ff89e28abf79f4730fced370fa50bb2af1
Oct 30 20:12:53.450405 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:53.450364    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v" podUID="815c2cbd-47e1-49ef-83cf-3f96ef171d74" containerName="payload-processing" containerID="cri-o://11dac1c7475d86cb0f4a2909b6bd2b55fc972a22dfcdac54473a91f7cddc6f11" gracePeriod=25
Oct 30 20:12:53.470759 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:53.470727    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99" podUID="00c823a0-8617-4d7c-8df8-8b69ff439424" containerName="payload-pre-processing" containerID="cri-o://e4d37713a969f0bd3edd2ab863c57374fb07ca0ea85e01dc6fcc9862f356617d" gracePeriod=25
Oct 30 20:12:54.047030 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:54.046980    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz" event={"ID":"7081733e-6a21-4108-bd76-40c7e4d98ea6","Type":"ContainerStarted","Data":"14c154fc891c20f1a5ddfeeae86e3e960f9644ae84bf17e423deff548d54bcd8"}
Oct 30 20:12:54.047030 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:54.047035    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz" event={"ID":"7081733e-6a21-4108-bd76-40c7e4d98ea6","Type":"ContainerStarted","Data":"6ebad7bb3b854238f8b730fa09bc9c40f17d41a3acbd48cef0f9c7483a8f0c9d"}
Oct 30 20:12:54.048773 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:54.048742    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5" event={"ID":"c6b6589b-3575-4241-820e-d17634751dac","Type":"ContainerStarted","Data":"b56825a4a29616189d82e606cc8a7d5dcd41373669781332b7d4bea9f6a927d5"}
Oct 30 20:12:54.048900 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:54.048797    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5" event={"ID":"c6b6589b-3575-4241-820e-d17634751dac","Type":"ContainerStarted","Data":"b906f78d6f91ceec62f52940fc2eb5ff89e28abf79f4730fced370fa50bb2af1"}
Oct 30 20:12:54.092647 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:54.092595    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz" podStartSLOduration=2.092577834 podStartE2EDuration="2.092577834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:54.086084545 +0000 UTC m=+2573.978698862" watchObservedRunningTime="2026-10-30 20:12:54.092577834 +0000 UTC m=+2573.985192161"
Oct 30 20:12:54.115604 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:54.115546    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5" podStartSLOduration=2.115529464 podStartE2EDuration="2.115529464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:54.114512418 +0000 UTC m=+2574.007126742" watchObservedRunningTime="2026-10-30 20:12:54.115529464 +0000 UTC m=+2574.008143789"
Oct 30 20:12:54.279043 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:54.278965    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52af467d-0e62-456d-b43c-016003f6e0b4" path="/var/lib/kubelet/pods/52af467d-0e62-456d-b43c-016003f6e0b4/volumes"
Oct 30 20:12:55.058558 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:55.058518    4823 generic.go:356] "Generic (PLEG): container finished" podID="7081733e-6a21-4108-bd76-40c7e4d98ea6" containerID="14c154fc891c20f1a5ddfeeae86e3e960f9644ae84bf17e423deff548d54bcd8" exitCode=0
Oct 30 20:12:55.059027 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:55.058600    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz" event={"ID":"7081733e-6a21-4108-bd76-40c7e4d98ea6","Type":"ContainerDied","Data":"14c154fc891c20f1a5ddfeeae86e3e960f9644ae84bf17e423deff548d54bcd8"}
Oct 30 20:12:55.060400 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:55.060375    4823 generic.go:356] "Generic (PLEG): container finished" podID="c6b6589b-3575-4241-820e-d17634751dac" containerID="b56825a4a29616189d82e606cc8a7d5dcd41373669781332b7d4bea9f6a927d5" exitCode=0
Oct 30 20:12:55.060540 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:55.060457    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5" event={"ID":"c6b6589b-3575-4241-820e-d17634751dac","Type":"ContainerDied","Data":"b56825a4a29616189d82e606cc8a7d5dcd41373669781332b7d4bea9f6a927d5"}
Oct 30 20:12:56.263424 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.263389    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"
Oct 30 20:12:56.333330 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.333291    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6b6589b-3575-4241-820e-d17634751dac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b6589b-3575-4241-820e-d17634751dac-maas-api-openshift-ca\") pod \"c6b6589b-3575-4241-820e-d17634751dac\" (UID: \"c6b6589b-3575-4241-820e-d17634751dac\") "
Oct 30 20:12:56.333781 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.333758    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b6589b-3575-4241-820e-d17634751dac-kube-api-access-nzrf7\" (UniqueName: \"kubernetes.io/projected/c6b6589b-3575-4241-820e-d17634751dac-kube-api-access-nzrf7\") pod \"c6b6589b-3575-4241-820e-d17634751dac\" (UID: \"c6b6589b-3575-4241-820e-d17634751dac\") "
Oct 30 20:12:56.334102 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.334083    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6b6589b-3575-4241-820e-d17634751dac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b6589b-3575-4241-820e-d17634751dac-maas-api-xks-ca\") pod \"c6b6589b-3575-4241-820e-d17634751dac\" (UID: \"c6b6589b-3575-4241-820e-d17634751dac\") "
Oct 30 20:12:56.336451 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.336425    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b6589b-3575-4241-820e-d17634751dac-maas-api-openshift-ca" pod "c6b6589b-3575-4241-820e-d17634751dac" (UID: "c6b6589b-3575-4241-820e-d17634751dac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:56.366755 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.366710    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6b6589b-3575-4241-820e-d17634751dac-maas-api-xks-ca" pod "c6b6589b-3575-4241-820e-d17634751dac" (UID: "c6b6589b-3575-4241-820e-d17634751dac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:56.367005 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.366976    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b6589b-3575-4241-820e-d17634751dac-kube-api-access-nzrf7" pod "c6b6589b-3575-4241-820e-d17634751dac" (UID: "c6b6589b-3575-4241-820e-d17634751dac"). InnerVolumeSpecName "kube-api-access-nzrf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:56.394130 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.394103    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"
Oct 30 20:12:56.436296 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.436264    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7081733e-6a21-4108-bd76-40c7e4d98ea6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7081733e-6a21-4108-bd76-40c7e4d98ea6-maas-api-openshift-ca\") pod \"7081733e-6a21-4108-bd76-40c7e4d98ea6\" (UID: \"7081733e-6a21-4108-bd76-40c7e4d98ea6\") "
Oct 30 20:12:56.436453 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.436409    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7081733e-6a21-4108-bd76-40c7e4d98ea6-kube-api-access-p4jtd\" (UniqueName: \"kubernetes.io/projected/7081733e-6a21-4108-bd76-40c7e4d98ea6-kube-api-access-p4jtd\") pod \"7081733e-6a21-4108-bd76-40c7e4d98ea6\" (UID: \"7081733e-6a21-4108-bd76-40c7e4d98ea6\") "
Oct 30 20:12:56.436517 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.436460    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7081733e-6a21-4108-bd76-40c7e4d98ea6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7081733e-6a21-4108-bd76-40c7e4d98ea6-maas-api-xks-ca\") pod \"7081733e-6a21-4108-bd76-40c7e4d98ea6\" (UID: \"7081733e-6a21-4108-bd76-40c7e4d98ea6\") "
Oct 30 20:12:56.436716 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.436687    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7081733e-6a21-4108-bd76-40c7e4d98ea6-maas-api-openshift-ca" pod "7081733e-6a21-4108-bd76-40c7e4d98ea6" (UID: "7081733e-6a21-4108-bd76-40c7e4d98ea6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:56.437042 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.436997    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzrf7\" (UniqueName: \"kubernetes.io/projected/c6b6589b-3575-4241-820e-d17634751dac-kube-api-access-nzrf7\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:56.437042 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.437027    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7081733e-6a21-4108-bd76-40c7e4d98ea6-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:56.437042 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.437044    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b6589b-3575-4241-820e-d17634751dac-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:56.437248 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.437055    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b6589b-3575-4241-820e-d17634751dac-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:56.439977 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.439945    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7081733e-6a21-4108-bd76-40c7e4d98ea6-maas-api-xks-ca" pod "7081733e-6a21-4108-bd76-40c7e4d98ea6" (UID: "7081733e-6a21-4108-bd76-40c7e4d98ea6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:56.440243 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.440212    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7081733e-6a21-4108-bd76-40c7e4d98ea6-kube-api-access-p4jtd" pod "7081733e-6a21-4108-bd76-40c7e4d98ea6" (UID: "7081733e-6a21-4108-bd76-40c7e4d98ea6"). InnerVolumeSpecName "kube-api-access-p4jtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:56.539007 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.538975    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4jtd\" (UniqueName: \"kubernetes.io/projected/7081733e-6a21-4108-bd76-40c7e4d98ea6-kube-api-access-p4jtd\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:56.539007 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:56.539003    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7081733e-6a21-4108-bd76-40c7e4d98ea6-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:57.072124 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:57.072082    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"]
Oct 30 20:12:57.077331 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:57.077307    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-278ffa-mw7fq"]
Oct 30 20:12:57.080853 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:57.080833    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"
Oct 30 20:12:57.080983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:57.080850    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz" event={"ID":"7081733e-6a21-4108-bd76-40c7e4d98ea6","Type":"ContainerDied","Data":"6ebad7bb3b854238f8b730fa09bc9c40f17d41a3acbd48cef0f9c7483a8f0c9d"}
Oct 30 20:12:57.080983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:57.080940    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ebad7bb3b854238f8b730fa09bc9c40f17d41a3acbd48cef0f9c7483a8f0c9d"
Oct 30 20:12:57.082359 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:57.082339    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"
Oct 30 20:12:57.082359 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:57.082344    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5" event={"ID":"c6b6589b-3575-4241-820e-d17634751dac","Type":"ContainerDied","Data":"b906f78d6f91ceec62f52940fc2eb5ff89e28abf79f4730fced370fa50bb2af1"}
Oct 30 20:12:57.082489 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:57.082392    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b906f78d6f91ceec62f52940fc2eb5ff89e28abf79f4730fced370fa50bb2af1"
Oct 30 20:12:58.249324 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.249276    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main" containerID="cri-o://88defcac7b38958448c575aefe3b2501c808a79c83cdba8c1d3376ec4916005a" gracePeriod=45
Oct 30 20:12:58.279316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.279288    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e8bbafa-1167-49d4-a51b-a90801d5d66b" path="/var/lib/kubelet/pods/4e8bbafa-1167-49d4-a51b-a90801d5d66b/volumes"
Oct 30 20:12:58.572316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.572284    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 20:12:58.668913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.668846    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-tmp-dir\") pod \"ca283777-7610-46f7-9fc2-cd13b31f7209\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") "
Oct 30 20:12:58.668913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.668909    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca283777-7610-46f7-9fc2-cd13b31f7209-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca283777-7610-46f7-9fc2-cd13b31f7209-tls-certs\") pod \"ca283777-7610-46f7-9fc2-cd13b31f7209\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") "
Oct 30 20:12:58.669188 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.668938    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-home\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-home\") pod \"ca283777-7610-46f7-9fc2-cd13b31f7209\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") "
Oct 30 20:12:58.669188 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.668972    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-model-cache\") pod \"ca283777-7610-46f7-9fc2-cd13b31f7209\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") "
Oct 30 20:12:58.669188 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.669133    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-tmp-dir" pod "ca283777-7610-46f7-9fc2-cd13b31f7209" (UID: "ca283777-7610-46f7-9fc2-cd13b31f7209"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:58.669303 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.669185    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca283777-7610-46f7-9fc2-cd13b31f7209-kube-api-access-2h5tf\" (UniqueName: \"kubernetes.io/projected/ca283777-7610-46f7-9fc2-cd13b31f7209-kube-api-access-2h5tf\") pod \"ca283777-7610-46f7-9fc2-cd13b31f7209\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") "
Oct 30 20:12:58.669303 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.669203    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-home" pod "ca283777-7610-46f7-9fc2-cd13b31f7209" (UID: "ca283777-7610-46f7-9fc2-cd13b31f7209"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:58.669303 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.669253    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-dshm\") pod \"ca283777-7610-46f7-9fc2-cd13b31f7209\" (UID: \"ca283777-7610-46f7-9fc2-cd13b31f7209\") "
Oct 30 20:12:58.669455 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.669349    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-model-cache" pod "ca283777-7610-46f7-9fc2-cd13b31f7209" (UID: "ca283777-7610-46f7-9fc2-cd13b31f7209"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:58.669676 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.669654    4823 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-tmp-dir\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:58.669676 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.669676    4823 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-home\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:58.669979 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.669685    4823 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-model-cache\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:58.671654 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.671626    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-dshm" pod "ca283777-7610-46f7-9fc2-cd13b31f7209" (UID: "ca283777-7610-46f7-9fc2-cd13b31f7209"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:58.671762 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.671625    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca283777-7610-46f7-9fc2-cd13b31f7209-tls-certs" pod "ca283777-7610-46f7-9fc2-cd13b31f7209" (UID: "ca283777-7610-46f7-9fc2-cd13b31f7209"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:58.671762 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.671671    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca283777-7610-46f7-9fc2-cd13b31f7209-kube-api-access-2h5tf" pod "ca283777-7610-46f7-9fc2-cd13b31f7209" (UID: "ca283777-7610-46f7-9fc2-cd13b31f7209"). InnerVolumeSpecName "kube-api-access-2h5tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:58.771151 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.771117    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h5tf\" (UniqueName: \"kubernetes.io/projected/ca283777-7610-46f7-9fc2-cd13b31f7209-kube-api-access-2h5tf\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:58.771151 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.771146    4823 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca283777-7610-46f7-9fc2-cd13b31f7209-dshm\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:58.771151 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:58.771157    4823 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca283777-7610-46f7-9fc2-cd13b31f7209-tls-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:59.106351 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:59.106315    4823 generic.go:356] "Generic (PLEG): container finished" podID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerID="88defcac7b38958448c575aefe3b2501c808a79c83cdba8c1d3376ec4916005a" exitCode=0
Oct 30 20:12:59.106553 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:59.106360    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"
Oct 30 20:12:59.106553 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:59.106400    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" event={"ID":"ca283777-7610-46f7-9fc2-cd13b31f7209","Type":"ContainerDied","Data":"88defcac7b38958448c575aefe3b2501c808a79c83cdba8c1d3376ec4916005a"}
Oct 30 20:12:59.106553 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:59.106443    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26" event={"ID":"ca283777-7610-46f7-9fc2-cd13b31f7209","Type":"ContainerDied","Data":"649acbce5cf6a411210dd6d602ed733b4806aef5db95a6cdfeb86ec68d8a631b"}
Oct 30 20:12:59.106553 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:59.106463    4823 scope.go:122] "RemoveContainer" containerID="88defcac7b38958448c575aefe3b2501c808a79c83cdba8c1d3376ec4916005a"
Oct 30 20:12:59.118192 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:59.118173    4823 scope.go:122] "RemoveContainer" containerID="88defcac7b38958448c575aefe3b2501c808a79c83cdba8c1d3376ec4916005a"
Oct 30 20:12:59.118441 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:12:59.118419    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88defcac7b38958448c575aefe3b2501c808a79c83cdba8c1d3376ec4916005a\": container with ID starting with 88defcac7b38958448c575aefe3b2501c808a79c83cdba8c1d3376ec4916005a not found: ID does not exist" containerID="88defcac7b38958448c575aefe3b2501c808a79c83cdba8c1d3376ec4916005a"
Oct 30 20:12:59.118495 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:59.118449    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88defcac7b38958448c575aefe3b2501c808a79c83cdba8c1d3376ec4916005a"} err="failed to get container status \"88defcac7b38958448c575aefe3b2501c808a79c83cdba8c1d3376ec4916005a\": rpc error: code = NotFound desc = could not find container \"88defcac7b38958448c575aefe3b2501c808a79c83cdba8c1d3376ec4916005a\": container with ID starting with 88defcac7b38958448c575aefe3b2501c808a79c83cdba8c1d3376ec4916005a not found: ID does not exist"
Oct 30 20:12:59.157961 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:59.157934    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"]
Oct 30 20:12:59.210544 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:12:59.210507    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/premium-w6-5f933e-kserve-55675c5fdf-z5r26"]
Oct 30 20:13:00.289326 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:00.289294    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" path="/var/lib/kubelet/pods/ca283777-7610-46f7-9fc2-cd13b31f7209/volumes"
Oct 30 20:13:00.385160 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:00.385116    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99" podUID="00c823a0-8617-4d7c-8df8-8b69ff439424" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:13:00.385160 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:00.385131    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v" podUID="815c2cbd-47e1-49ef-83cf-3f96ef171d74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:13:02.015422 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:13:02.015383    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:13:07.783321 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:07.783283    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"]
Oct 30 20:13:07.783741 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:07.783515    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm" podUID="82f08656-55bf-4de5-9beb-3776bc8dd3f2" containerName="istio-proxy" containerID="cri-o://dfad4186f2aeae3d8c24ddf6a94b9c5999592637abc0be617df6e7f3a0ed4a02" gracePeriod=30
Oct 30 20:13:09.818125 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:09.818074    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"]
Oct 30 20:13:09.818961 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:09.818915    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl" podUID="fa894be2-b885-4001-912e-f6996634431f" containerName="istio-proxy" containerID="cri-o://bcfcf3a305c97bba0cd57e7e390c2b238567407929c693487f13a1faa5d78869" gracePeriod=30
Oct 30 20:13:09.824693 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:09.824665    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"]
Oct 30 20:13:09.825010 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:09.824966    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm" podUID="ad11ffea-d313-4369-81f8-70014b50603f" containerName="istio-proxy" containerID="cri-o://f870e6a685e7ce309cabb73665dfaddc0de0e9e39ad63245a98ccb6100e17358" gracePeriod=30
Oct 30 20:13:10.384788 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:10.384754    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99" podUID="00c823a0-8617-4d7c-8df8-8b69ff439424" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:13:10.384978 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:10.384795    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v" podUID="815c2cbd-47e1-49ef-83cf-3f96ef171d74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:13:11.159289 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:11.159256    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"]
Oct 30 20:13:11.159701 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:11.159466    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerName="istio-proxy" containerID="cri-o://9b06bfcedddd5e9f159e43c5a14b2b0abfdf388dcd76852891dad656bafa45a3" gracePeriod=30
Oct 30 20:13:13.105646 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.105554    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"]
Oct 30 20:13:13.111388 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.111360    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5f933e-wlrs4"]
Oct 30 20:13:13.181827 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.181805    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 20:13:13.255617 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.255561    4823 generic.go:356] "Generic (PLEG): container finished" podID="82f08656-55bf-4de5-9beb-3776bc8dd3f2" containerID="dfad4186f2aeae3d8c24ddf6a94b9c5999592637abc0be617df6e7f3a0ed4a02" exitCode=0
Oct 30 20:13:13.255826 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.255727    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm" event={"ID":"82f08656-55bf-4de5-9beb-3776bc8dd3f2","Type":"ContainerDied","Data":"dfad4186f2aeae3d8c24ddf6a94b9c5999592637abc0be617df6e7f3a0ed4a02"}
Oct 30 20:13:13.255951 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.255827    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm" event={"ID":"82f08656-55bf-4de5-9beb-3776bc8dd3f2","Type":"ContainerDied","Data":"2995c2de98c2851b089489e5db0c5549a99bf6d9cf7b3fa7f000b7045cfd9cd8"}
Oct 30 20:13:13.255951 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.255882    4823 scope.go:122] "RemoveContainer" containerID="dfad4186f2aeae3d8c24ddf6a94b9c5999592637abc0be617df6e7f3a0ed4a02"
Oct 30 20:13:13.256115 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.256098    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"
Oct 30 20:13:13.277248 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.277230    4823 scope.go:122] "RemoveContainer" containerID="2208641275ce75519737c1cd9455c82a180c032775afbfcf7b62c1002c93a7e6"
Oct 30 20:13:13.290557 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.290525    4823 scope.go:122] "RemoveContainer" containerID="dfad4186f2aeae3d8c24ddf6a94b9c5999592637abc0be617df6e7f3a0ed4a02"
Oct 30 20:13:13.290933 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:13:13.290904    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfad4186f2aeae3d8c24ddf6a94b9c5999592637abc0be617df6e7f3a0ed4a02\": container with ID starting with dfad4186f2aeae3d8c24ddf6a94b9c5999592637abc0be617df6e7f3a0ed4a02 not found: ID does not exist" containerID="dfad4186f2aeae3d8c24ddf6a94b9c5999592637abc0be617df6e7f3a0ed4a02"
Oct 30 20:13:13.291054 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.290944    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfad4186f2aeae3d8c24ddf6a94b9c5999592637abc0be617df6e7f3a0ed4a02"} err="failed to get container status \"dfad4186f2aeae3d8c24ddf6a94b9c5999592637abc0be617df6e7f3a0ed4a02\": rpc error: code = NotFound desc = could not find container \"dfad4186f2aeae3d8c24ddf6a94b9c5999592637abc0be617df6e7f3a0ed4a02\": container with ID starting with dfad4186f2aeae3d8c24ddf6a94b9c5999592637abc0be617df6e7f3a0ed4a02 not found: ID does not exist"
Oct 30 20:13:13.291054 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.290973    4823 scope.go:122] "RemoveContainer" containerID="2208641275ce75519737c1cd9455c82a180c032775afbfcf7b62c1002c93a7e6"
Oct 30 20:13:13.291270 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:13:13.291245    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2208641275ce75519737c1cd9455c82a180c032775afbfcf7b62c1002c93a7e6\": container with ID starting with 2208641275ce75519737c1cd9455c82a180c032775afbfcf7b62c1002c93a7e6 not found: ID does not exist" containerID="2208641275ce75519737c1cd9455c82a180c032775afbfcf7b62c1002c93a7e6"
Oct 30 20:13:13.291314 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.291279    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2208641275ce75519737c1cd9455c82a180c032775afbfcf7b62c1002c93a7e6"} err="failed to get container status \"2208641275ce75519737c1cd9455c82a180c032775afbfcf7b62c1002c93a7e6\": rpc error: code = NotFound desc = could not find container \"2208641275ce75519737c1cd9455c82a180c032775afbfcf7b62c1002c93a7e6\": container with ID starting with 2208641275ce75519737c1cd9455c82a180c032775afbfcf7b62c1002c93a7e6 not found: ID does not exist"
Oct 30 20:13:13.328938 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.328904    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-workload-certs\") pod \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") "
Oct 30 20:13:13.329114 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.328990    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-credential-socket\") pod \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") "
Oct 30 20:13:13.329114 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.329070    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-envoy\") pod \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") "
Oct 30 20:13:13.329256 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.329110    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-workload-certs" pod "82f08656-55bf-4de5-9beb-3776bc8dd3f2" (UID: "82f08656-55bf-4de5-9beb-3776bc8dd3f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.329256 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.329142    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-token\" (UniqueName: \"kubernetes.io/projected/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-token\") pod \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") "
Oct 30 20:13:13.329256 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.329198    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-data\") pod \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") "
Oct 30 20:13:13.329256 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.329231    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istiod-ca-cert\") pod \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") "
Oct 30 20:13:13.329488 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.329296    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-podinfo\") pod \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") "
Oct 30 20:13:13.329488 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.329339    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-credential-socket" pod "82f08656-55bf-4de5-9beb-3776bc8dd3f2" (UID: "82f08656-55bf-4de5-9beb-3776bc8dd3f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.329488 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.329415    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-workload-socket\") pod \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") "
Oct 30 20:13:13.329488 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.329473    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f08656-55bf-4de5-9beb-3776bc8dd3f2-kube-api-access-4t9pr\" (UniqueName: \"kubernetes.io/projected/82f08656-55bf-4de5-9beb-3776bc8dd3f2-kube-api-access-4t9pr\") pod \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\" (UID: \"82f08656-55bf-4de5-9beb-3776bc8dd3f2\") "
Oct 30 20:13:13.329837 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.329551    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-data" pod "82f08656-55bf-4de5-9beb-3776bc8dd3f2" (UID: "82f08656-55bf-4de5-9beb-3776bc8dd3f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.329837 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.329681    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istiod-ca-cert" pod "82f08656-55bf-4de5-9beb-3776bc8dd3f2" (UID: "82f08656-55bf-4de5-9beb-3776bc8dd3f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:13.330273 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.330245    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-workload-socket" pod "82f08656-55bf-4de5-9beb-3776bc8dd3f2" (UID: "82f08656-55bf-4de5-9beb-3776bc8dd3f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.330273 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.330257    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.330425 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.330296    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.330425 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.330316    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.330425 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.330331    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.332191 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.332164    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f08656-55bf-4de5-9beb-3776bc8dd3f2-kube-api-access-4t9pr" pod "82f08656-55bf-4de5-9beb-3776bc8dd3f2" (UID: "82f08656-55bf-4de5-9beb-3776bc8dd3f2"). InnerVolumeSpecName "kube-api-access-4t9pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:13.332298 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.332280    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-token" pod "82f08656-55bf-4de5-9beb-3776bc8dd3f2" (UID: "82f08656-55bf-4de5-9beb-3776bc8dd3f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:13.332747 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.332730    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-envoy" pod "82f08656-55bf-4de5-9beb-3776bc8dd3f2" (UID: "82f08656-55bf-4de5-9beb-3776bc8dd3f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:13.333130 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.333098    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-podinfo" pod "82f08656-55bf-4de5-9beb-3776bc8dd3f2" (UID: "82f08656-55bf-4de5-9beb-3776bc8dd3f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:13:13.432030 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.431993    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.432030 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.432025    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.432239 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.432039    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9pr\" (UniqueName: \"kubernetes.io/projected/82f08656-55bf-4de5-9beb-3776bc8dd3f2-kube-api-access-4t9pr\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.432239 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.432053    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.432239 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.432064    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f08656-55bf-4de5-9beb-3776bc8dd3f2-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.596806 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.596778    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"]
Oct 30 20:13:13.600604 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.600572    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-5s4rm"]
Oct 30 20:13:13.743077 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.743047    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"
Oct 30 20:13:13.810307 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.810271    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"
Oct 30 20:13:13.837162 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.837135    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/815c2cbd-47e1-49ef-83cf-3f96ef171d74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/815c2cbd-47e1-49ef-83cf-3f96ef171d74-plugins-config-volume\") pod \"815c2cbd-47e1-49ef-83cf-3f96ef171d74\" (UID: \"815c2cbd-47e1-49ef-83cf-3f96ef171d74\") "
Oct 30 20:13:13.837353 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.837337    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815c2cbd-47e1-49ef-83cf-3f96ef171d74-kube-api-access-cvpb4\" (UniqueName: \"kubernetes.io/projected/815c2cbd-47e1-49ef-83cf-3f96ef171d74-kube-api-access-cvpb4\") pod \"815c2cbd-47e1-49ef-83cf-3f96ef171d74\" (UID: \"815c2cbd-47e1-49ef-83cf-3f96ef171d74\") "
Oct 30 20:13:13.837551 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.837528    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/815c2cbd-47e1-49ef-83cf-3f96ef171d74-plugins-config-volume" pod "815c2cbd-47e1-49ef-83cf-3f96ef171d74" (UID: "815c2cbd-47e1-49ef-83cf-3f96ef171d74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:13.837801 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.837783    4823 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/815c2cbd-47e1-49ef-83cf-3f96ef171d74-plugins-config-volume\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.839579 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.839560    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815c2cbd-47e1-49ef-83cf-3f96ef171d74-kube-api-access-cvpb4" pod "815c2cbd-47e1-49ef-83cf-3f96ef171d74" (UID: "815c2cbd-47e1-49ef-83cf-3f96ef171d74"). InnerVolumeSpecName "kube-api-access-cvpb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:13.939367 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.939337    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c823a0-8617-4d7c-8df8-8b69ff439424-kube-api-access-n8p66\" (UniqueName: \"kubernetes.io/projected/00c823a0-8617-4d7c-8df8-8b69ff439424-kube-api-access-n8p66\") pod \"00c823a0-8617-4d7c-8df8-8b69ff439424\" (UID: \"00c823a0-8617-4d7c-8df8-8b69ff439424\") "
Oct 30 20:13:13.939564 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.939454    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c823a0-8617-4d7c-8df8-8b69ff439424-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c823a0-8617-4d7c-8df8-8b69ff439424-plugins-config-volume\") pod \"00c823a0-8617-4d7c-8df8-8b69ff439424\" (UID: \"00c823a0-8617-4d7c-8df8-8b69ff439424\") "
Oct 30 20:13:13.939798 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.939782    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvpb4\" (UniqueName: \"kubernetes.io/projected/815c2cbd-47e1-49ef-83cf-3f96ef171d74-kube-api-access-cvpb4\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:13.939915 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.939788    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c823a0-8617-4d7c-8df8-8b69ff439424-plugins-config-volume" pod "00c823a0-8617-4d7c-8df8-8b69ff439424" (UID: "00c823a0-8617-4d7c-8df8-8b69ff439424"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:13.941745 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:13.941719    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c823a0-8617-4d7c-8df8-8b69ff439424-kube-api-access-n8p66" pod "00c823a0-8617-4d7c-8df8-8b69ff439424" (UID: "00c823a0-8617-4d7c-8df8-8b69ff439424"). InnerVolumeSpecName "kube-api-access-n8p66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:14.041608 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.041523    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8p66\" (UniqueName: \"kubernetes.io/projected/00c823a0-8617-4d7c-8df8-8b69ff439424-kube-api-access-n8p66\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:14.041608 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.041554    4823 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c823a0-8617-4d7c-8df8-8b69ff439424-plugins-config-volume\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:14.265067 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.265030    4823 generic.go:356] "Generic (PLEG): container finished" podID="815c2cbd-47e1-49ef-83cf-3f96ef171d74" containerID="11dac1c7475d86cb0f4a2909b6bd2b55fc972a22dfcdac54473a91f7cddc6f11" exitCode=0
Oct 30 20:13:14.265502 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.265089    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"
Oct 30 20:13:14.265502 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.265105    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v" event={"ID":"815c2cbd-47e1-49ef-83cf-3f96ef171d74","Type":"ContainerDied","Data":"11dac1c7475d86cb0f4a2909b6bd2b55fc972a22dfcdac54473a91f7cddc6f11"}
Oct 30 20:13:14.265502 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.265155    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v" event={"ID":"815c2cbd-47e1-49ef-83cf-3f96ef171d74","Type":"ContainerDied","Data":"d8e2d5cb7711c1d8e8edcb01166e1da295944f1402b8c2ad445d7b992cd9ccc8"}
Oct 30 20:13:14.265502 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.265186    4823 scope.go:122] "RemoveContainer" containerID="11dac1c7475d86cb0f4a2909b6bd2b55fc972a22dfcdac54473a91f7cddc6f11"
Oct 30 20:13:14.266897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.266856    4823 generic.go:356] "Generic (PLEG): container finished" podID="00c823a0-8617-4d7c-8df8-8b69ff439424" containerID="e4d37713a969f0bd3edd2ab863c57374fb07ca0ea85e01dc6fcc9862f356617d" exitCode=0
Oct 30 20:13:14.267043 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.266956    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99" event={"ID":"00c823a0-8617-4d7c-8df8-8b69ff439424","Type":"ContainerDied","Data":"e4d37713a969f0bd3edd2ab863c57374fb07ca0ea85e01dc6fcc9862f356617d"}
Oct 30 20:13:14.267043 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.266973    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"
Oct 30 20:13:14.267043 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.267006    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99" event={"ID":"00c823a0-8617-4d7c-8df8-8b69ff439424","Type":"ContainerDied","Data":"122dcec6a637b8ec1aa2ec3fb65a4fec793867d6f2f26e7436bc347276a0982a"}
Oct 30 20:13:14.276633 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.276617    4823 scope.go:122] "RemoveContainer" containerID="11dac1c7475d86cb0f4a2909b6bd2b55fc972a22dfcdac54473a91f7cddc6f11"
Oct 30 20:13:14.276941 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:13:14.276913    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11dac1c7475d86cb0f4a2909b6bd2b55fc972a22dfcdac54473a91f7cddc6f11\": container with ID starting with 11dac1c7475d86cb0f4a2909b6bd2b55fc972a22dfcdac54473a91f7cddc6f11 not found: ID does not exist" containerID="11dac1c7475d86cb0f4a2909b6bd2b55fc972a22dfcdac54473a91f7cddc6f11"
Oct 30 20:13:14.277032 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.276953    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dac1c7475d86cb0f4a2909b6bd2b55fc972a22dfcdac54473a91f7cddc6f11"} err="failed to get container status \"11dac1c7475d86cb0f4a2909b6bd2b55fc972a22dfcdac54473a91f7cddc6f11\": rpc error: code = NotFound desc = could not find container \"11dac1c7475d86cb0f4a2909b6bd2b55fc972a22dfcdac54473a91f7cddc6f11\": container with ID starting with 11dac1c7475d86cb0f4a2909b6bd2b55fc972a22dfcdac54473a91f7cddc6f11 not found: ID does not exist"
Oct 30 20:13:14.277032 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.276977    4823 scope.go:122] "RemoveContainer" containerID="e4d37713a969f0bd3edd2ab863c57374fb07ca0ea85e01dc6fcc9862f356617d"
Oct 30 20:13:14.280314 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.280292    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b5959c-301e-4849-91c3-f8ffecd21731" path="/var/lib/kubelet/pods/71b5959c-301e-4849-91c3-f8ffecd21731/volumes"
Oct 30 20:13:14.280805 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.280790    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f08656-55bf-4de5-9beb-3776bc8dd3f2" path="/var/lib/kubelet/pods/82f08656-55bf-4de5-9beb-3776bc8dd3f2/volumes"
Oct 30 20:13:14.323125 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.323003    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"]
Oct 30 20:13:14.324118 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.324095    4823 scope.go:122] "RemoveContainer" containerID="e4d37713a969f0bd3edd2ab863c57374fb07ca0ea85e01dc6fcc9862f356617d"
Oct 30 20:13:14.324452 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:13:14.324431    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d37713a969f0bd3edd2ab863c57374fb07ca0ea85e01dc6fcc9862f356617d\": container with ID starting with e4d37713a969f0bd3edd2ab863c57374fb07ca0ea85e01dc6fcc9862f356617d not found: ID does not exist" containerID="e4d37713a969f0bd3edd2ab863c57374fb07ca0ea85e01dc6fcc9862f356617d"
Oct 30 20:13:14.324541 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.324460    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d37713a969f0bd3edd2ab863c57374fb07ca0ea85e01dc6fcc9862f356617d"} err="failed to get container status \"e4d37713a969f0bd3edd2ab863c57374fb07ca0ea85e01dc6fcc9862f356617d\": rpc error: code = NotFound desc = could not find container \"e4d37713a969f0bd3edd2ab863c57374fb07ca0ea85e01dc6fcc9862f356617d\": container with ID starting with e4d37713a969f0bd3edd2ab863c57374fb07ca0ea85e01dc6fcc9862f356617d not found: ID does not exist"
Oct 30 20:13:14.328479 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.328457    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5f933e-6554df5d4-fnl4v"]
Oct 30 20:13:14.332985 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.332956    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"]
Oct 30 20:13:14.339634 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.339611    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5f933e-6ddf77b97f-pcs99"]
Oct 30 20:13:14.784386 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.784350    4823 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:13:14.784558 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:14.784413    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:13:15.098523 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.098493    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 20:13:15.155253 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.155229    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 20:13:15.259057 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259021    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-workload-socket\") pod \"fa894be2-b885-4001-912e-f6996634431f\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") "
Oct 30 20:13:15.259057 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259063    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad11ffea-d313-4369-81f8-70014b50603f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad11ffea-d313-4369-81f8-70014b50603f-istio-podinfo\") pod \"ad11ffea-d313-4369-81f8-70014b50603f\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") "
Oct 30 20:13:15.259288 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259103    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-workload-socket\") pod \"ad11ffea-d313-4369-81f8-70014b50603f\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") "
Oct 30 20:13:15.259288 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259135    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa894be2-b885-4001-912e-f6996634431f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa894be2-b885-4001-912e-f6996634431f-istiod-ca-cert\") pod \"fa894be2-b885-4001-912e-f6996634431f\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") "
Oct 30 20:13:15.259288 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259179    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-workload-certs\") pod \"ad11ffea-d313-4369-81f8-70014b50603f\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") "
Oct 30 20:13:15.259288 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259208    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa894be2-b885-4001-912e-f6996634431f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa894be2-b885-4001-912e-f6996634431f-istio-podinfo\") pod \"fa894be2-b885-4001-912e-f6996634431f\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") "
Oct 30 20:13:15.259288 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259236    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-istio-envoy\") pod \"fa894be2-b885-4001-912e-f6996634431f\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") "
Oct 30 20:13:15.259587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259299    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-workload-certs\") pod \"fa894be2-b885-4001-912e-f6996634431f\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") "
Oct 30 20:13:15.259587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259340    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad11ffea-d313-4369-81f8-70014b50603f-istio-token\" (UniqueName: \"kubernetes.io/projected/ad11ffea-d313-4369-81f8-70014b50603f-istio-token\") pod \"ad11ffea-d313-4369-81f8-70014b50603f\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") "
Oct 30 20:13:15.259587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259371    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-credential-socket\") pod \"ad11ffea-d313-4369-81f8-70014b50603f\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") "
Oct 30 20:13:15.259587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259361    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-workload-socket" pod "fa894be2-b885-4001-912e-f6996634431f" (UID: "fa894be2-b885-4001-912e-f6996634431f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.259587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259420    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad11ffea-d313-4369-81f8-70014b50603f-kube-api-access-fkpkn\" (UniqueName: \"kubernetes.io/projected/ad11ffea-d313-4369-81f8-70014b50603f-kube-api-access-fkpkn\") pod \"ad11ffea-d313-4369-81f8-70014b50603f\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") "
Oct 30 20:13:15.259587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259441    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-workload-socket" pod "ad11ffea-d313-4369-81f8-70014b50603f" (UID: "ad11ffea-d313-4369-81f8-70014b50603f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.259587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259490    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-istio-envoy\") pod \"ad11ffea-d313-4369-81f8-70014b50603f\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") "
Oct 30 20:13:15.259587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259529    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa894be2-b885-4001-912e-f6996634431f-istio-token\" (UniqueName: \"kubernetes.io/projected/fa894be2-b885-4001-912e-f6996634431f-istio-token\") pod \"fa894be2-b885-4001-912e-f6996634431f\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") "
Oct 30 20:13:15.259587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259550    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-workload-certs" pod "ad11ffea-d313-4369-81f8-70014b50603f" (UID: "ad11ffea-d313-4369-81f8-70014b50603f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.261408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259589    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa894be2-b885-4001-912e-f6996634431f-istiod-ca-cert" pod "fa894be2-b885-4001-912e-f6996634431f" (UID: "fa894be2-b885-4001-912e-f6996634431f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:15.261408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259617    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-workload-certs" pod "fa894be2-b885-4001-912e-f6996634431f" (UID: "fa894be2-b885-4001-912e-f6996634431f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.261408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259638    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-istio-data\") pod \"fa894be2-b885-4001-912e-f6996634431f\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") "
Oct 30 20:13:15.261408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259684    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-istio-data\") pod \"ad11ffea-d313-4369-81f8-70014b50603f\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") "
Oct 30 20:13:15.261408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259728    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad11ffea-d313-4369-81f8-70014b50603f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad11ffea-d313-4369-81f8-70014b50603f-istiod-ca-cert\") pod \"ad11ffea-d313-4369-81f8-70014b50603f\" (UID: \"ad11ffea-d313-4369-81f8-70014b50603f\") "
Oct 30 20:13:15.261408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259793    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-credential-socket" pod "ad11ffea-d313-4369-81f8-70014b50603f" (UID: "ad11ffea-d313-4369-81f8-70014b50603f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.261408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259840    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-credential-socket\") pod \"fa894be2-b885-4001-912e-f6996634431f\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") "
Oct 30 20:13:15.261408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.259907    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa894be2-b885-4001-912e-f6996634431f-kube-api-access-6sn89\" (UniqueName: \"kubernetes.io/projected/fa894be2-b885-4001-912e-f6996634431f-kube-api-access-6sn89\") pod \"fa894be2-b885-4001-912e-f6996634431f\" (UID: \"fa894be2-b885-4001-912e-f6996634431f\") "
Oct 30 20:13:15.261408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.260086    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-istio-data" pod "fa894be2-b885-4001-912e-f6996634431f" (UID: "fa894be2-b885-4001-912e-f6996634431f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.261408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.260570    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.261408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.260593    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.261408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.260610    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.261408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.260640    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa894be2-b885-4001-912e-f6996634431f-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.261408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.260653    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.261408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.260667    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.261408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.260682    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.262430 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.262405    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa894be2-b885-4001-912e-f6996634431f-istio-podinfo" pod "fa894be2-b885-4001-912e-f6996634431f" (UID: "fa894be2-b885-4001-912e-f6996634431f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:13:15.262504 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.262482    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-istio-envoy" pod "ad11ffea-d313-4369-81f8-70014b50603f" (UID: "ad11ffea-d313-4369-81f8-70014b50603f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.262637 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.262610    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-istio-envoy" pod "fa894be2-b885-4001-912e-f6996634431f" (UID: "fa894be2-b885-4001-912e-f6996634431f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.262728 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.262644    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad11ffea-d313-4369-81f8-70014b50603f-kube-api-access-fkpkn" pod "ad11ffea-d313-4369-81f8-70014b50603f" (UID: "ad11ffea-d313-4369-81f8-70014b50603f"). InnerVolumeSpecName "kube-api-access-fkpkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:15.262791 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.262729    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa894be2-b885-4001-912e-f6996634431f-kube-api-access-6sn89" pod "fa894be2-b885-4001-912e-f6996634431f" (UID: "fa894be2-b885-4001-912e-f6996634431f"). InnerVolumeSpecName "kube-api-access-6sn89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:15.262791 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.262754    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-istio-data" pod "ad11ffea-d313-4369-81f8-70014b50603f" (UID: "ad11ffea-d313-4369-81f8-70014b50603f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.262951 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.262931    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-credential-socket" pod "fa894be2-b885-4001-912e-f6996634431f" (UID: "fa894be2-b885-4001-912e-f6996634431f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:15.263042 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.262960    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad11ffea-d313-4369-81f8-70014b50603f-istiod-ca-cert" pod "ad11ffea-d313-4369-81f8-70014b50603f" (UID: "ad11ffea-d313-4369-81f8-70014b50603f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:15.263042 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.263013    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa894be2-b885-4001-912e-f6996634431f-istio-token" pod "fa894be2-b885-4001-912e-f6996634431f" (UID: "fa894be2-b885-4001-912e-f6996634431f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:15.263305 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.263290    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad11ffea-d313-4369-81f8-70014b50603f-istio-token" pod "ad11ffea-d313-4369-81f8-70014b50603f" (UID: "ad11ffea-d313-4369-81f8-70014b50603f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:15.263480 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.263463    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad11ffea-d313-4369-81f8-70014b50603f-istio-podinfo" pod "ad11ffea-d313-4369-81f8-70014b50603f" (UID: "ad11ffea-d313-4369-81f8-70014b50603f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:13:15.279249 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.279220    4823 generic.go:356] "Generic (PLEG): container finished" podID="fa894be2-b885-4001-912e-f6996634431f" containerID="bcfcf3a305c97bba0cd57e7e390c2b238567407929c693487f13a1faa5d78869" exitCode=0
Oct 30 20:13:15.279646 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.279260    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"
Oct 30 20:13:15.279646 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.279296    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl" event={"ID":"fa894be2-b885-4001-912e-f6996634431f","Type":"ContainerDied","Data":"bcfcf3a305c97bba0cd57e7e390c2b238567407929c693487f13a1faa5d78869"}
Oct 30 20:13:15.279646 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.279340    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl" event={"ID":"fa894be2-b885-4001-912e-f6996634431f","Type":"ContainerDied","Data":"852dfc64046f10bb3738a0a5e3277e57f8a44b3d68b189e56cc5849107d1bc53"}
Oct 30 20:13:15.279646 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.279368    4823 scope.go:122] "RemoveContainer" containerID="bcfcf3a305c97bba0cd57e7e390c2b238567407929c693487f13a1faa5d78869"
Oct 30 20:13:15.280757 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.280734    4823 generic.go:356] "Generic (PLEG): container finished" podID="ad11ffea-d313-4369-81f8-70014b50603f" containerID="f870e6a685e7ce309cabb73665dfaddc0de0e9e39ad63245a98ccb6100e17358" exitCode=0
Oct 30 20:13:15.280846 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.280772    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"
Oct 30 20:13:15.280846 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.280825    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm" event={"ID":"ad11ffea-d313-4369-81f8-70014b50603f","Type":"ContainerDied","Data":"f870e6a685e7ce309cabb73665dfaddc0de0e9e39ad63245a98ccb6100e17358"}
Oct 30 20:13:15.281002 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.280883    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm" event={"ID":"ad11ffea-d313-4369-81f8-70014b50603f","Type":"ContainerDied","Data":"55d4f6c0d4f597a9890dfa7107369ab3be5590a7375f0a38f3a3561a6195bdd7"}
Oct 30 20:13:15.292378 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.292358    4823 scope.go:122] "RemoveContainer" containerID="bcfcf3a305c97bba0cd57e7e390c2b238567407929c693487f13a1faa5d78869"
Oct 30 20:13:15.292678 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:13:15.292660    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcfcf3a305c97bba0cd57e7e390c2b238567407929c693487f13a1faa5d78869\": container with ID starting with bcfcf3a305c97bba0cd57e7e390c2b238567407929c693487f13a1faa5d78869 not found: ID does not exist" containerID="bcfcf3a305c97bba0cd57e7e390c2b238567407929c693487f13a1faa5d78869"
Oct 30 20:13:15.292728 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.292685    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcfcf3a305c97bba0cd57e7e390c2b238567407929c693487f13a1faa5d78869"} err="failed to get container status \"bcfcf3a305c97bba0cd57e7e390c2b238567407929c693487f13a1faa5d78869\": rpc error: code = NotFound desc = could not find container \"bcfcf3a305c97bba0cd57e7e390c2b238567407929c693487f13a1faa5d78869\": container with ID starting with bcfcf3a305c97bba0cd57e7e390c2b238567407929c693487f13a1faa5d78869 not found: ID does not exist"
Oct 30 20:13:15.292728 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.292699    4823 scope.go:122] "RemoveContainer" containerID="f870e6a685e7ce309cabb73665dfaddc0de0e9e39ad63245a98ccb6100e17358"
Oct 30 20:13:15.303464 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.303447    4823 scope.go:122] "RemoveContainer" containerID="f870e6a685e7ce309cabb73665dfaddc0de0e9e39ad63245a98ccb6100e17358"
Oct 30 20:13:15.303722 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:13:15.303700    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f870e6a685e7ce309cabb73665dfaddc0de0e9e39ad63245a98ccb6100e17358\": container with ID starting with f870e6a685e7ce309cabb73665dfaddc0de0e9e39ad63245a98ccb6100e17358 not found: ID does not exist" containerID="f870e6a685e7ce309cabb73665dfaddc0de0e9e39ad63245a98ccb6100e17358"
Oct 30 20:13:15.303771 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.303731    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f870e6a685e7ce309cabb73665dfaddc0de0e9e39ad63245a98ccb6100e17358"} err="failed to get container status \"f870e6a685e7ce309cabb73665dfaddc0de0e9e39ad63245a98ccb6100e17358\": rpc error: code = NotFound desc = could not find container \"f870e6a685e7ce309cabb73665dfaddc0de0e9e39ad63245a98ccb6100e17358\": container with ID starting with f870e6a685e7ce309cabb73665dfaddc0de0e9e39ad63245a98ccb6100e17358 not found: ID does not exist"
Oct 30 20:13:15.312165 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.312110    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"]
Oct 30 20:13:15.320378 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.320358    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-f8dxl"]
Oct 30 20:13:15.332271 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.332239    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"]
Oct 30 20:13:15.335702 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.335677    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-j72gm"]
Oct 30 20:13:15.361976 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.361946    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkpkn\" (UniqueName: \"kubernetes.io/projected/ad11ffea-d313-4369-81f8-70014b50603f-kube-api-access-fkpkn\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.361976 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.361977    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.362200 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.361994    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa894be2-b885-4001-912e-f6996634431f-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.362200 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.362011    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad11ffea-d313-4369-81f8-70014b50603f-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.362200 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.362026    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad11ffea-d313-4369-81f8-70014b50603f-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.362200 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.362041    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.362200 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.362056    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sn89\" (UniqueName: \"kubernetes.io/projected/fa894be2-b885-4001-912e-f6996634431f-kube-api-access-6sn89\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.362200 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.362070    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad11ffea-d313-4369-81f8-70014b50603f-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.362200 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.362086    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa894be2-b885-4001-912e-f6996634431f-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.362200 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.362104    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa894be2-b885-4001-912e-f6996634431f-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:15.362200 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:15.362120    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad11ffea-d313-4369-81f8-70014b50603f-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.285104 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.285059    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c823a0-8617-4d7c-8df8-8b69ff439424" path="/var/lib/kubelet/pods/00c823a0-8617-4d7c-8df8-8b69ff439424/volumes"
Oct 30 20:13:16.285769 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.285741    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="815c2cbd-47e1-49ef-83cf-3f96ef171d74" path="/var/lib/kubelet/pods/815c2cbd-47e1-49ef-83cf-3f96ef171d74/volumes"
Oct 30 20:13:16.286471 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.286449    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad11ffea-d313-4369-81f8-70014b50603f" path="/var/lib/kubelet/pods/ad11ffea-d313-4369-81f8-70014b50603f/volumes"
Oct 30 20:13:16.287954 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.287927    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa894be2-b885-4001-912e-f6996634431f" path="/var/lib/kubelet/pods/fa894be2-b885-4001-912e-f6996634431f/volumes"
Oct 30 20:13:16.294456 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.294434    4823 generic.go:356] "Generic (PLEG): container finished" podID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerID="9b06bfcedddd5e9f159e43c5a14b2b0abfdf388dcd76852891dad656bafa45a3" exitCode=0
Oct 30 20:13:16.294574 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.294507    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" event={"ID":"c965af44-94c0-4f66-8905-7d86c8b1f546","Type":"ContainerDied","Data":"9b06bfcedddd5e9f159e43c5a14b2b0abfdf388dcd76852891dad656bafa45a3"}
Oct 30 20:13:16.294574 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.294571    4823 scope.go:122] "RemoveContainer" containerID="0d4eff7631177d4ce9d48c8b7eb6b76a1a16b4989f9fb844c877477a941380c9"
Oct 30 20:13:16.451258 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.451224    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 20:13:16.579296 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.579267    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-workload-certs\") pod \"c965af44-94c0-4f66-8905-7d86c8b1f546\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") "
Oct 30 20:13:16.579492 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.579321    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-envoy\") pod \"c965af44-94c0-4f66-8905-7d86c8b1f546\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") "
Oct 30 20:13:16.579492 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.579370    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-token\" (UniqueName: \"kubernetes.io/projected/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-token\") pod \"c965af44-94c0-4f66-8905-7d86c8b1f546\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") "
Oct 30 20:13:16.579492 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.579422    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-credential-socket\") pod \"c965af44-94c0-4f66-8905-7d86c8b1f546\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") "
Oct 30 20:13:16.579492 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.579459    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-podinfo\") pod \"c965af44-94c0-4f66-8905-7d86c8b1f546\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") "
Oct 30 20:13:16.579744 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.579516    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-workload-socket\") pod \"c965af44-94c0-4f66-8905-7d86c8b1f546\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") "
Oct 30 20:13:16.579744 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.579543    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c965af44-94c0-4f66-8905-7d86c8b1f546-kube-api-access-nsfcf\" (UniqueName: \"kubernetes.io/projected/c965af44-94c0-4f66-8905-7d86c8b1f546-kube-api-access-nsfcf\") pod \"c965af44-94c0-4f66-8905-7d86c8b1f546\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") "
Oct 30 20:13:16.579744 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.579615    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-workload-certs" pod "c965af44-94c0-4f66-8905-7d86c8b1f546" (UID: "c965af44-94c0-4f66-8905-7d86c8b1f546"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:16.579744 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.579677    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c965af44-94c0-4f66-8905-7d86c8b1f546-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c965af44-94c0-4f66-8905-7d86c8b1f546-istiod-ca-cert\") pod \"c965af44-94c0-4f66-8905-7d86c8b1f546\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") "
Oct 30 20:13:16.579985 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.579756    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-credential-socket" pod "c965af44-94c0-4f66-8905-7d86c8b1f546" (UID: "c965af44-94c0-4f66-8905-7d86c8b1f546"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:16.579985 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.579764    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-data\") pod \"c965af44-94c0-4f66-8905-7d86c8b1f546\" (UID: \"c965af44-94c0-4f66-8905-7d86c8b1f546\") "
Oct 30 20:13:16.579985 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.579853    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-workload-socket" pod "c965af44-94c0-4f66-8905-7d86c8b1f546" (UID: "c965af44-94c0-4f66-8905-7d86c8b1f546"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:16.580092 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.579979    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-data" pod "c965af44-94c0-4f66-8905-7d86c8b1f546" (UID: "c965af44-94c0-4f66-8905-7d86c8b1f546"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:16.580129 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.580113    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c965af44-94c0-4f66-8905-7d86c8b1f546-istiod-ca-cert" pod "c965af44-94c0-4f66-8905-7d86c8b1f546" (UID: "c965af44-94c0-4f66-8905-7d86c8b1f546"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:16.580397 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.580379    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c965af44-94c0-4f66-8905-7d86c8b1f546-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.580397 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.580401    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.580516 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.580410    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.580516 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.580419    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.580516 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.580428    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.582136 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.582110    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c965af44-94c0-4f66-8905-7d86c8b1f546-kube-api-access-nsfcf" pod "c965af44-94c0-4f66-8905-7d86c8b1f546" (UID: "c965af44-94c0-4f66-8905-7d86c8b1f546"). InnerVolumeSpecName "kube-api-access-nsfcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:16.582238 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.582195    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-envoy" pod "c965af44-94c0-4f66-8905-7d86c8b1f546" (UID: "c965af44-94c0-4f66-8905-7d86c8b1f546"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:16.582291 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.582249    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-podinfo" pod "c965af44-94c0-4f66-8905-7d86c8b1f546" (UID: "c965af44-94c0-4f66-8905-7d86c8b1f546"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:13:16.582326 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.582306    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-token" pod "c965af44-94c0-4f66-8905-7d86c8b1f546" (UID: "c965af44-94c0-4f66-8905-7d86c8b1f546"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:16.681415 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.681336    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.681415 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.681367    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.681415 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.681378    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c965af44-94c0-4f66-8905-7d86c8b1f546-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:16.681415 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:16.681388    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsfcf\" (UniqueName: \"kubernetes.io/projected/c965af44-94c0-4f66-8905-7d86c8b1f546-kube-api-access-nsfcf\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:17.306206 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:17.306172    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"
Oct 30 20:13:17.306705 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:17.306218    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6" event={"ID":"c965af44-94c0-4f66-8905-7d86c8b1f546","Type":"ContainerDied","Data":"cb8f279463b5a29a9b923409a3272fb8ffb9c14a184658e774b223cb92742e5e"}
Oct 30 20:13:17.306705 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:17.306282    4823 scope.go:122] "RemoveContainer" containerID="9b06bfcedddd5e9f159e43c5a14b2b0abfdf388dcd76852891dad656bafa45a3"
Oct 30 20:13:17.335698 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:17.335672    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"]
Oct 30 20:13:17.348290 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:17.348266    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-zjtc6"]
Oct 30 20:13:18.280618 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:18.280581    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" path="/var/lib/kubelet/pods/c965af44-94c0-4f66-8905-7d86c8b1f546/volumes"
Oct 30 20:13:22.846097 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:22.846056    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"]
Oct 30 20:13:22.846570 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:22.846314    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt" podUID="a247c39a-03e3-4f42-a939-3c80ecc3bff5" containerName="istio-proxy" containerID="cri-o://5b30dde630cc917ab7e0fb595d4359e80163a118eb824b7a33d5cfb1994fc87f" gracePeriod=30
Oct 30 20:13:22.850954 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:22.850158    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"]
Oct 30 20:13:22.850954 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:22.850396    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj" podUID="f7954659-6a11-45ba-80e1-af0684ae90a3" containerName="istio-proxy" containerID="cri-o://86722aa378879ef0acba47fea75d9ad0214951be34d48030850f5e42449b71aa" gracePeriod=30
Oct 30 20:13:28.129895 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.129798    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 20:13:28.154454 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.154419    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 20:13:28.252306 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.252274    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-credential-socket\") pod \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") "
Oct 30 20:13:28.252465 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.252323    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-credential-socket\") pod \"f7954659-6a11-45ba-80e1-af0684ae90a3\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") "
Oct 30 20:13:28.252465 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.252370    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-workload-socket\") pod \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") "
Oct 30 20:13:28.252465 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.252452    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7954659-6a11-45ba-80e1-af0684ae90a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7954659-6a11-45ba-80e1-af0684ae90a3-istiod-ca-cert\") pod \"f7954659-6a11-45ba-80e1-af0684ae90a3\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") "
Oct 30 20:13:28.252597 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.252485    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-data\") pod \"f7954659-6a11-45ba-80e1-af0684ae90a3\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") "
Oct 30 20:13:28.252597 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.252538    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-podinfo\") pod \"f7954659-6a11-45ba-80e1-af0684ae90a3\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") "
Oct 30 20:13:28.252597 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.252591    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-envoy\") pod \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") "
Oct 30 20:13:28.252765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.252631    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-envoy\") pod \"f7954659-6a11-45ba-80e1-af0684ae90a3\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") "
Oct 30 20:13:28.252765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.252630    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-credential-socket" pod "a247c39a-03e3-4f42-a939-3c80ecc3bff5" (UID: "a247c39a-03e3-4f42-a939-3c80ecc3bff5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:28.252765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.252666    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-token\" (UniqueName: \"kubernetes.io/projected/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-token\") pod \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") "
Oct 30 20:13:28.252765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.252672    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-credential-socket" pod "f7954659-6a11-45ba-80e1-af0684ae90a3" (UID: "f7954659-6a11-45ba-80e1-af0684ae90a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:28.252765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.252716    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-token\" (UniqueName: \"kubernetes.io/projected/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-token\") pod \"f7954659-6a11-45ba-80e1-af0684ae90a3\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") "
Oct 30 20:13:28.252765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.252719    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-workload-socket" pod "a247c39a-03e3-4f42-a939-3c80ecc3bff5" (UID: "a247c39a-03e3-4f42-a939-3c80ecc3bff5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:28.252765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.252750    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a247c39a-03e3-4f42-a939-3c80ecc3bff5-kube-api-access-r8nrz\" (UniqueName: \"kubernetes.io/projected/a247c39a-03e3-4f42-a939-3c80ecc3bff5-kube-api-access-r8nrz\") pod \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") "
Oct 30 20:13:28.253207 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.252786    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7954659-6a11-45ba-80e1-af0684ae90a3-kube-api-access-wwc2p\" (UniqueName: \"kubernetes.io/projected/f7954659-6a11-45ba-80e1-af0684ae90a3-kube-api-access-wwc2p\") pod \"f7954659-6a11-45ba-80e1-af0684ae90a3\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") "
Oct 30 20:13:28.253207 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.252850    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7954659-6a11-45ba-80e1-af0684ae90a3-istiod-ca-cert" pod "f7954659-6a11-45ba-80e1-af0684ae90a3" (UID: "f7954659-6a11-45ba-80e1-af0684ae90a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:28.253207 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.253176    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-data\") pod \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") "
Oct 30 20:13:28.253388 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.253210    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-data" pod "f7954659-6a11-45ba-80e1-af0684ae90a3" (UID: "f7954659-6a11-45ba-80e1-af0684ae90a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:28.253388 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.253240    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-workload-certs\") pod \"f7954659-6a11-45ba-80e1-af0684ae90a3\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") "
Oct 30 20:13:28.253388 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.253280    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-workload-socket\") pod \"f7954659-6a11-45ba-80e1-af0684ae90a3\" (UID: \"f7954659-6a11-45ba-80e1-af0684ae90a3\") "
Oct 30 20:13:28.253388 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.253324    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-workload-certs\") pod \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") "
Oct 30 20:13:28.253899 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.253494    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istiod-ca-cert\") pod \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") "
Oct 30 20:13:28.253899 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.253527    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-podinfo\") pod \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\" (UID: \"a247c39a-03e3-4f42-a939-3c80ecc3bff5\") "
Oct 30 20:13:28.253899 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.253540    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-data" pod "a247c39a-03e3-4f42-a939-3c80ecc3bff5" (UID: "a247c39a-03e3-4f42-a939-3c80ecc3bff5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:28.253899 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.253733    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-workload-socket" pod "f7954659-6a11-45ba-80e1-af0684ae90a3" (UID: "f7954659-6a11-45ba-80e1-af0684ae90a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:28.254161 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.253946    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-workload-certs" pod "f7954659-6a11-45ba-80e1-af0684ae90a3" (UID: "f7954659-6a11-45ba-80e1-af0684ae90a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:28.254246 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.254187    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istiod-ca-cert" pod "a247c39a-03e3-4f42-a939-3c80ecc3bff5" (UID: "a247c39a-03e3-4f42-a939-3c80ecc3bff5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:28.254368 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.254334    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-workload-certs" pod "a247c39a-03e3-4f42-a939-3c80ecc3bff5" (UID: "a247c39a-03e3-4f42-a939-3c80ecc3bff5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:28.254816 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.254793    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7954659-6a11-45ba-80e1-af0684ae90a3-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.254958 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.254821    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.254958 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.254836    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.254958 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.254849    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.254958 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.254915    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.254958 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.254933    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.254958 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.254947    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.255413 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.254962    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.255413 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.254977    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.255413 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.254993    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.255588 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.255506    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-envoy" pod "a247c39a-03e3-4f42-a939-3c80ecc3bff5" (UID: "a247c39a-03e3-4f42-a939-3c80ecc3bff5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:28.255588 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.255509    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-envoy" pod "f7954659-6a11-45ba-80e1-af0684ae90a3" (UID: "f7954659-6a11-45ba-80e1-af0684ae90a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:28.255765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.255744    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7954659-6a11-45ba-80e1-af0684ae90a3-kube-api-access-wwc2p" pod "f7954659-6a11-45ba-80e1-af0684ae90a3" (UID: "f7954659-6a11-45ba-80e1-af0684ae90a3"). InnerVolumeSpecName "kube-api-access-wwc2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:28.255900 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.255873    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-podinfo" pod "f7954659-6a11-45ba-80e1-af0684ae90a3" (UID: "f7954659-6a11-45ba-80e1-af0684ae90a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:13:28.256191 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.256162    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-podinfo" pod "a247c39a-03e3-4f42-a939-3c80ecc3bff5" (UID: "a247c39a-03e3-4f42-a939-3c80ecc3bff5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:13:28.256363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.256343    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-token" pod "f7954659-6a11-45ba-80e1-af0684ae90a3" (UID: "f7954659-6a11-45ba-80e1-af0684ae90a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:28.256947 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.256928    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-token" pod "a247c39a-03e3-4f42-a939-3c80ecc3bff5" (UID: "a247c39a-03e3-4f42-a939-3c80ecc3bff5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:28.257033 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.257011    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a247c39a-03e3-4f42-a939-3c80ecc3bff5-kube-api-access-r8nrz" pod "a247c39a-03e3-4f42-a939-3c80ecc3bff5" (UID: "a247c39a-03e3-4f42-a939-3c80ecc3bff5"). InnerVolumeSpecName "kube-api-access-r8nrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:28.356400 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.356369    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.356400 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.356402    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8nrz\" (UniqueName: \"kubernetes.io/projected/a247c39a-03e3-4f42-a939-3c80ecc3bff5-kube-api-access-r8nrz\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.356574 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.356418    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwc2p\" (UniqueName: \"kubernetes.io/projected/f7954659-6a11-45ba-80e1-af0684ae90a3-kube-api-access-wwc2p\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.356574 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.356433    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.356574 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.356448    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.356574 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.356463    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.356574 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.356477    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7954659-6a11-45ba-80e1-af0684ae90a3-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.356574 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.356492    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a247c39a-03e3-4f42-a939-3c80ecc3bff5-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:28.434649 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.434553    4823 generic.go:356] "Generic (PLEG): container finished" podID="a247c39a-03e3-4f42-a939-3c80ecc3bff5" containerID="5b30dde630cc917ab7e0fb595d4359e80163a118eb824b7a33d5cfb1994fc87f" exitCode=0
Oct 30 20:13:28.434649 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.434583    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt" event={"ID":"a247c39a-03e3-4f42-a939-3c80ecc3bff5","Type":"ContainerDied","Data":"5b30dde630cc917ab7e0fb595d4359e80163a118eb824b7a33d5cfb1994fc87f"}
Oct 30 20:13:28.434649 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.434628    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt" event={"ID":"a247c39a-03e3-4f42-a939-3c80ecc3bff5","Type":"ContainerDied","Data":"21c60ccc9aa26bd0081ca2f2a0136671d719898670c5c2fc405c8091139bca18"}
Oct 30 20:13:28.434649 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.434567    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"
Oct 30 20:13:28.434649 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.434651    4823 scope.go:122] "RemoveContainer" containerID="5b30dde630cc917ab7e0fb595d4359e80163a118eb824b7a33d5cfb1994fc87f"
Oct 30 20:13:28.436656 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.436576    4823 generic.go:356] "Generic (PLEG): container finished" podID="f7954659-6a11-45ba-80e1-af0684ae90a3" containerID="86722aa378879ef0acba47fea75d9ad0214951be34d48030850f5e42449b71aa" exitCode=0
Oct 30 20:13:28.436656 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.436639    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj" event={"ID":"f7954659-6a11-45ba-80e1-af0684ae90a3","Type":"ContainerDied","Data":"86722aa378879ef0acba47fea75d9ad0214951be34d48030850f5e42449b71aa"}
Oct 30 20:13:28.436845 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.436677    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj" event={"ID":"f7954659-6a11-45ba-80e1-af0684ae90a3","Type":"ContainerDied","Data":"4ac965aa3c67d73f22c6010947ac2996109dd74fb835bc511784d08f5434be3c"}
Oct 30 20:13:28.437553 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.436632    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"
Oct 30 20:13:28.447626 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.447604    4823 scope.go:122] "RemoveContainer" containerID="5b30dde630cc917ab7e0fb595d4359e80163a118eb824b7a33d5cfb1994fc87f"
Oct 30 20:13:28.447967 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:13:28.447945    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b30dde630cc917ab7e0fb595d4359e80163a118eb824b7a33d5cfb1994fc87f\": container with ID starting with 5b30dde630cc917ab7e0fb595d4359e80163a118eb824b7a33d5cfb1994fc87f not found: ID does not exist" containerID="5b30dde630cc917ab7e0fb595d4359e80163a118eb824b7a33d5cfb1994fc87f"
Oct 30 20:13:28.448033 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.447974    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b30dde630cc917ab7e0fb595d4359e80163a118eb824b7a33d5cfb1994fc87f"} err="failed to get container status \"5b30dde630cc917ab7e0fb595d4359e80163a118eb824b7a33d5cfb1994fc87f\": rpc error: code = NotFound desc = could not find container \"5b30dde630cc917ab7e0fb595d4359e80163a118eb824b7a33d5cfb1994fc87f\": container with ID starting with 5b30dde630cc917ab7e0fb595d4359e80163a118eb824b7a33d5cfb1994fc87f not found: ID does not exist"
Oct 30 20:13:28.448033 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.447990    4823 scope.go:122] "RemoveContainer" containerID="86722aa378879ef0acba47fea75d9ad0214951be34d48030850f5e42449b71aa"
Oct 30 20:13:28.457815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.457798    4823 scope.go:122] "RemoveContainer" containerID="86722aa378879ef0acba47fea75d9ad0214951be34d48030850f5e42449b71aa"
Oct 30 20:13:28.458133 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:13:28.458105    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86722aa378879ef0acba47fea75d9ad0214951be34d48030850f5e42449b71aa\": container with ID starting with 86722aa378879ef0acba47fea75d9ad0214951be34d48030850f5e42449b71aa not found: ID does not exist" containerID="86722aa378879ef0acba47fea75d9ad0214951be34d48030850f5e42449b71aa"
Oct 30 20:13:28.458202 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.458141    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86722aa378879ef0acba47fea75d9ad0214951be34d48030850f5e42449b71aa"} err="failed to get container status \"86722aa378879ef0acba47fea75d9ad0214951be34d48030850f5e42449b71aa\": rpc error: code = NotFound desc = could not find container \"86722aa378879ef0acba47fea75d9ad0214951be34d48030850f5e42449b71aa\": container with ID starting with 86722aa378879ef0acba47fea75d9ad0214951be34d48030850f5e42449b71aa not found: ID does not exist"
Oct 30 20:13:28.466945 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.466925    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"]
Oct 30 20:13:28.472128 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.472108    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-87ljt"]
Oct 30 20:13:28.499997 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.499964    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"]
Oct 30 20:13:28.502923 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:28.502905    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-jdpcj"]
Oct 30 20:13:30.280629 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:30.280592    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a247c39a-03e3-4f42-a939-3c80ecc3bff5" path="/var/lib/kubelet/pods/a247c39a-03e3-4f42-a939-3c80ecc3bff5/volumes"
Oct 30 20:13:30.281095 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:30.281080    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7954659-6a11-45ba-80e1-af0684ae90a3" path="/var/lib/kubelet/pods/f7954659-6a11-45ba-80e1-af0684ae90a3/volumes"
Oct 30 20:13:32.014602 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:13:32.014562    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:13:38.235936 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:38.235897    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"]
Oct 30 20:13:38.236358 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:38.236140    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg" podUID="ee23fe73-0972-42d0-b734-f4c3748354f9" containerName="istio-proxy" containerID="cri-o://b4765da50ae3065bada0594f2ed7ea89c389721c857a6ca03eecc337ee1caff7" gracePeriod=30
Oct 30 20:13:40.557357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:40.557326    4823 generic.go:356] "Generic (PLEG): container finished" podID="778e153e-48af-4716-9fb2-2095e02c7bc2" containerID="a13408882b277f94c3e5ec6cc9f708b7f0d754940aa41676980403d80cfa3938" exitCode=0
Oct 30 20:13:40.557830 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:40.557398    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25" event={"ID":"778e153e-48af-4716-9fb2-2095e02c7bc2","Type":"ContainerDied","Data":"a13408882b277f94c3e5ec6cc9f708b7f0d754940aa41676980403d80cfa3938"}
Oct 30 20:13:40.557830 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:40.557800    4823 scope.go:122] "RemoveContainer" containerID="a13408882b277f94c3e5ec6cc9f708b7f0d754940aa41676980403d80cfa3938"
Oct 30 20:13:40.559135 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:40.559103    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:13:40.559249 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:40.559184    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:13:40.559249 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:40.559232    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:13:41.606114 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:41.605306    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25" event={"ID":"778e153e-48af-4716-9fb2-2095e02c7bc2","Type":"ContainerStarted","Data":"78552bdf49410b8849d4f370fe26108fc01e2dbcecd7ca43f12084f2f14c4628"}
Oct 30 20:13:42.101194 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:42.101107    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 20:13:42.101363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:42.101326    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:13:42.101408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:42.101375    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25" podUID="778e153e-48af-4716-9fb2-2095e02c7bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 20:13:43.101830 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.101795    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:13:43.102273 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.101893    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25" podUID="778e153e-48af-4716-9fb2-2095e02c7bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 20:13:43.550154 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.550127    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 20:13:43.623767 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.623723    4823 generic.go:356] "Generic (PLEG): container finished" podID="ee23fe73-0972-42d0-b734-f4c3748354f9" containerID="b4765da50ae3065bada0594f2ed7ea89c389721c857a6ca03eecc337ee1caff7" exitCode=0
Oct 30 20:13:43.623767 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.623771    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"
Oct 30 20:13:43.624031 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.623806    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg" event={"ID":"ee23fe73-0972-42d0-b734-f4c3748354f9","Type":"ContainerDied","Data":"b4765da50ae3065bada0594f2ed7ea89c389721c857a6ca03eecc337ee1caff7"}
Oct 30 20:13:43.624031 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.623885    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg" event={"ID":"ee23fe73-0972-42d0-b734-f4c3748354f9","Type":"ContainerDied","Data":"6440e6754345281c6f03af7d72a64fdaf9008354d696396924daca6ff4a339b2"}
Oct 30 20:13:43.624031 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.623920    4823 scope.go:122] "RemoveContainer" containerID="b4765da50ae3065bada0594f2ed7ea89c389721c857a6ca03eecc337ee1caff7"
Oct 30 20:13:43.638003 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.637980    4823 scope.go:122] "RemoveContainer" containerID="2ab25446eb3edfaf9652e1e8ae0dd54962f42ce73fd1b25bd157f97198a30115"
Oct 30 20:13:43.649199 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.649177    4823 scope.go:122] "RemoveContainer" containerID="b4765da50ae3065bada0594f2ed7ea89c389721c857a6ca03eecc337ee1caff7"
Oct 30 20:13:43.649496 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:13:43.649469    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4765da50ae3065bada0594f2ed7ea89c389721c857a6ca03eecc337ee1caff7\": container with ID starting with b4765da50ae3065bada0594f2ed7ea89c389721c857a6ca03eecc337ee1caff7 not found: ID does not exist" containerID="b4765da50ae3065bada0594f2ed7ea89c389721c857a6ca03eecc337ee1caff7"
Oct 30 20:13:43.649557 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.649505    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4765da50ae3065bada0594f2ed7ea89c389721c857a6ca03eecc337ee1caff7"} err="failed to get container status \"b4765da50ae3065bada0594f2ed7ea89c389721c857a6ca03eecc337ee1caff7\": rpc error: code = NotFound desc = could not find container \"b4765da50ae3065bada0594f2ed7ea89c389721c857a6ca03eecc337ee1caff7\": container with ID starting with b4765da50ae3065bada0594f2ed7ea89c389721c857a6ca03eecc337ee1caff7 not found: ID does not exist"
Oct 30 20:13:43.649557 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.649525    4823 scope.go:122] "RemoveContainer" containerID="2ab25446eb3edfaf9652e1e8ae0dd54962f42ce73fd1b25bd157f97198a30115"
Oct 30 20:13:43.649771 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:13:43.649755    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab25446eb3edfaf9652e1e8ae0dd54962f42ce73fd1b25bd157f97198a30115\": container with ID starting with 2ab25446eb3edfaf9652e1e8ae0dd54962f42ce73fd1b25bd157f97198a30115 not found: ID does not exist" containerID="2ab25446eb3edfaf9652e1e8ae0dd54962f42ce73fd1b25bd157f97198a30115"
Oct 30 20:13:43.649822 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.649774    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab25446eb3edfaf9652e1e8ae0dd54962f42ce73fd1b25bd157f97198a30115"} err="failed to get container status \"2ab25446eb3edfaf9652e1e8ae0dd54962f42ce73fd1b25bd157f97198a30115\": rpc error: code = NotFound desc = could not find container \"2ab25446eb3edfaf9652e1e8ae0dd54962f42ce73fd1b25bd157f97198a30115\": container with ID starting with 2ab25446eb3edfaf9652e1e8ae0dd54962f42ce73fd1b25bd157f97198a30115 not found: ID does not exist"
Oct 30 20:13:43.681400 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.681360    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-credential-socket\") pod \"ee23fe73-0972-42d0-b734-f4c3748354f9\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") "
Oct 30 20:13:43.681594 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.681445    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee23fe73-0972-42d0-b734-f4c3748354f9-kube-api-access-gv9mj\" (UniqueName: \"kubernetes.io/projected/ee23fe73-0972-42d0-b734-f4c3748354f9-kube-api-access-gv9mj\") pod \"ee23fe73-0972-42d0-b734-f4c3748354f9\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") "
Oct 30 20:13:43.681594 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.681487    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-podinfo\") pod \"ee23fe73-0972-42d0-b734-f4c3748354f9\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") "
Oct 30 20:13:43.681594 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.681539    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-envoy\") pod \"ee23fe73-0972-42d0-b734-f4c3748354f9\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") "
Oct 30 20:13:43.681594 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.681565    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-workload-certs\") pod \"ee23fe73-0972-42d0-b734-f4c3748354f9\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") "
Oct 30 20:13:43.681825 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.681607    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-data\") pod \"ee23fe73-0972-42d0-b734-f4c3748354f9\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") "
Oct 30 20:13:43.681825 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.681638    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-workload-socket\") pod \"ee23fe73-0972-42d0-b734-f4c3748354f9\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") "
Oct 30 20:13:43.681825 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.681718    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-credential-socket" pod "ee23fe73-0972-42d0-b734-f4c3748354f9" (UID: "ee23fe73-0972-42d0-b734-f4c3748354f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:43.682041 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.681899    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee23fe73-0972-42d0-b734-f4c3748354f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee23fe73-0972-42d0-b734-f4c3748354f9-istiod-ca-cert\") pod \"ee23fe73-0972-42d0-b734-f4c3748354f9\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") "
Oct 30 20:13:43.682041 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.681934    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-token\" (UniqueName: \"kubernetes.io/projected/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-token\") pod \"ee23fe73-0972-42d0-b734-f4c3748354f9\" (UID: \"ee23fe73-0972-42d0-b734-f4c3748354f9\") "
Oct 30 20:13:43.682157 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.681896    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-workload-certs" pod "ee23fe73-0972-42d0-b734-f4c3748354f9" (UID: "ee23fe73-0972-42d0-b734-f4c3748354f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:43.682157 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.682124    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-workload-socket" pod "ee23fe73-0972-42d0-b734-f4c3748354f9" (UID: "ee23fe73-0972-42d0-b734-f4c3748354f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:43.682267 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.682059    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-data" pod "ee23fe73-0972-42d0-b734-f4c3748354f9" (UID: "ee23fe73-0972-42d0-b734-f4c3748354f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:43.682330 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.682267    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee23fe73-0972-42d0-b734-f4c3748354f9-istiod-ca-cert" pod "ee23fe73-0972-42d0-b734-f4c3748354f9" (UID: "ee23fe73-0972-42d0-b734-f4c3748354f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:43.682604 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.682576    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:43.682691 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.682608    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:43.682691 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.682623    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:43.682691 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.682639    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee23fe73-0972-42d0-b734-f4c3748354f9-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:43.682691 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.682653    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:43.684293 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.684247    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-envoy" pod "ee23fe73-0972-42d0-b734-f4c3748354f9" (UID: "ee23fe73-0972-42d0-b734-f4c3748354f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:43.684293 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.684278    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-podinfo" pod "ee23fe73-0972-42d0-b734-f4c3748354f9" (UID: "ee23fe73-0972-42d0-b734-f4c3748354f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:13:43.684454 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.684365    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-token" pod "ee23fe73-0972-42d0-b734-f4c3748354f9" (UID: "ee23fe73-0972-42d0-b734-f4c3748354f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:43.684454 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.684442    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee23fe73-0972-42d0-b734-f4c3748354f9-kube-api-access-gv9mj" pod "ee23fe73-0972-42d0-b734-f4c3748354f9" (UID: "ee23fe73-0972-42d0-b734-f4c3748354f9"). InnerVolumeSpecName "kube-api-access-gv9mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:43.784056 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.784021    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv9mj\" (UniqueName: \"kubernetes.io/projected/ee23fe73-0972-42d0-b734-f4c3748354f9-kube-api-access-gv9mj\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:43.784056 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.784052    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:43.784056 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.784061    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:43.784302 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.784071    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee23fe73-0972-42d0-b734-f4c3748354f9-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:43.824301 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.824270    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"]
Oct 30 20:13:43.835182 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.835153    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"]
Oct 30 20:13:43.983757 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.983714    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"]
Oct 30 20:13:43.990202 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:43.990169    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-3d0a78-openshift-default-f9f6d664c-5hdqg"]
Oct 30 20:13:44.101607 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:44.101557    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:13:44.101844 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:44.101628    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25" podUID="778e153e-48af-4716-9fb2-2095e02c7bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 20:13:44.284973 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:44.284884    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee23fe73-0972-42d0-b734-f4c3748354f9" path="/var/lib/kubelet/pods/ee23fe73-0972-42d0-b734-f4c3748354f9/volumes"
Oct 30 20:13:45.101374 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:45.101340    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:13:45.101576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:45.101393    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25" podUID="778e153e-48af-4716-9fb2-2095e02c7bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 20:13:46.101749 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:46.101711    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:13:46.102186 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:46.101768    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25" podUID="778e153e-48af-4716-9fb2-2095e02c7bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 20:13:46.701305 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:46.701266    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 20:13:47.105265 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:47.105198    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 20:13:47.141423 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:47.141378    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:13:47.663455 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:47.663428    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 20:13:52.482128 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482086    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"]
Oct 30 20:13:52.482655 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482640    4823 cpu_manager.go:414] "containerMap: removing container" podUID="ee23fe73-0972-42d0-b734-f4c3748354f9" containerName="istio-proxy"
Oct 30 20:13:52.482701 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482656    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee23fe73-0972-42d0-b734-f4c3748354f9" containerName="istio-proxy"
Oct 30 20:13:52.482701 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482674    4823 cpu_manager.go:414] "containerMap: removing container" podUID="815c2cbd-47e1-49ef-83cf-3f96ef171d74" containerName="payload-processing"
Oct 30 20:13:52.482701 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482679    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815c2cbd-47e1-49ef-83cf-3f96ef171d74" containerName="payload-processing"
Oct 30 20:13:52.482701 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482692    4823 cpu_manager.go:414] "containerMap: removing container" podUID="00c823a0-8617-4d7c-8df8-8b69ff439424" containerName="payload-pre-processing"
Oct 30 20:13:52.482701 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482697    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c823a0-8617-4d7c-8df8-8b69ff439424" containerName="payload-pre-processing"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482706    4823 cpu_manager.go:414] "containerMap: removing container" podUID="f7954659-6a11-45ba-80e1-af0684ae90a3" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482711    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7954659-6a11-45ba-80e1-af0684ae90a3" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482730    4823 cpu_manager.go:414] "containerMap: removing container" podUID="ad11ffea-d313-4369-81f8-70014b50603f" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482735    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad11ffea-d313-4369-81f8-70014b50603f" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482742    4823 cpu_manager.go:414] "containerMap: removing container" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482746    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482752    4823 cpu_manager.go:414] "containerMap: removing container" podUID="c6b6589b-3575-4241-820e-d17634751dac" containerName="revoke-keys"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482756    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b6589b-3575-4241-820e-d17634751dac" containerName="revoke-keys"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482764    4823 cpu_manager.go:414] "containerMap: removing container" podUID="ee23fe73-0972-42d0-b734-f4c3748354f9" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482768    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee23fe73-0972-42d0-b734-f4c3748354f9" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482778    4823 cpu_manager.go:414] "containerMap: removing container" podUID="82f08656-55bf-4de5-9beb-3776bc8dd3f2" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482783    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f08656-55bf-4de5-9beb-3776bc8dd3f2" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482803    4823 cpu_manager.go:414] "containerMap: removing container" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482807    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482815    4823 cpu_manager.go:414] "containerMap: removing container" podUID="a247c39a-03e3-4f42-a939-3c80ecc3bff5" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482819    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a247c39a-03e3-4f42-a939-3c80ecc3bff5" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482831    4823 cpu_manager.go:414] "containerMap: removing container" podUID="82f08656-55bf-4de5-9beb-3776bc8dd3f2" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482835    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f08656-55bf-4de5-9beb-3776bc8dd3f2" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482842    4823 cpu_manager.go:414] "containerMap: removing container" podUID="fa894be2-b885-4001-912e-f6996634431f" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482845    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa894be2-b885-4001-912e-f6996634431f" containerName="istio-proxy"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482872    4823 cpu_manager.go:414] "containerMap: removing container" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482876    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482883    4823 cpu_manager.go:414] "containerMap: removing container" podUID="7081733e-6a21-4108-bd76-40c7e4d98ea6" containerName="revoke-keys"
Oct 30 20:13:52.482910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482891    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7081733e-6a21-4108-bd76-40c7e4d98ea6" containerName="revoke-keys"
Oct 30 20:13:52.483628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.482994    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c823a0-8617-4d7c-8df8-8b69ff439424" containerName="payload-pre-processing"
Oct 30 20:13:52.483628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.483002    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7954659-6a11-45ba-80e1-af0684ae90a3" containerName="istio-proxy"
Oct 30 20:13:52.483628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.483010    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad11ffea-d313-4369-81f8-70014b50603f" containerName="istio-proxy"
Oct 30 20:13:52.483628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.483015    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerName="istio-proxy"
Oct 30 20:13:52.483628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.483022    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b6589b-3575-4241-820e-d17634751dac" containerName="revoke-keys"
Oct 30 20:13:52.483628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.483028    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee23fe73-0972-42d0-b734-f4c3748354f9" containerName="istio-proxy"
Oct 30 20:13:52.483628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.483034    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f08656-55bf-4de5-9beb-3776bc8dd3f2" containerName="istio-proxy"
Oct 30 20:13:52.483628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.483040    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="c965af44-94c0-4f66-8905-7d86c8b1f546" containerName="istio-proxy"
Oct 30 20:13:52.483628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.483045    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="a247c39a-03e3-4f42-a939-3c80ecc3bff5" containerName="istio-proxy"
Oct 30 20:13:52.483628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.483050    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f08656-55bf-4de5-9beb-3776bc8dd3f2" containerName="istio-proxy"
Oct 30 20:13:52.483628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.483055    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa894be2-b885-4001-912e-f6996634431f" containerName="istio-proxy"
Oct 30 20:13:52.483628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.483061    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca283777-7610-46f7-9fc2-cd13b31f7209" containerName="main"
Oct 30 20:13:52.483628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.483067    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="7081733e-6a21-4108-bd76-40c7e4d98ea6" containerName="revoke-keys"
Oct 30 20:13:52.483628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.483074    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="815c2cbd-47e1-49ef-83cf-3f96ef171d74" containerName="payload-processing"
Oct 30 20:13:52.488223 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.488205    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"
Oct 30 20:13:52.493468 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.493420    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:13:52.493655 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.493432    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:13:52.493655 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.493438    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:13:52.493815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.493467    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:13:52.500207 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.498775    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"]
Oct 30 20:13:52.602397 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.602364    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pgff\" (UniqueName: \"kubernetes.io/projected/fd839149-8249-44f4-baf7-9fcedbb58670-kube-api-access-8pgff\") pod \"maas-api-revoke-sub-f75163a5-8hx99\" (UID: \"fd839149-8249-44f4-baf7-9fcedbb58670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"
Oct 30 20:13:52.602602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.602501    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd839149-8249-44f4-baf7-9fcedbb58670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f75163a5-8hx99\" (UID: \"fd839149-8249-44f4-baf7-9fcedbb58670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"
Oct 30 20:13:52.602676 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.602656    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd839149-8249-44f4-baf7-9fcedbb58670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f75163a5-8hx99\" (UID: \"fd839149-8249-44f4-baf7-9fcedbb58670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"
Oct 30 20:13:52.703772 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.703729    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd839149-8249-44f4-baf7-9fcedbb58670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f75163a5-8hx99\" (UID: \"fd839149-8249-44f4-baf7-9fcedbb58670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"
Oct 30 20:13:52.704006 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.703821    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pgff\" (UniqueName: \"kubernetes.io/projected/fd839149-8249-44f4-baf7-9fcedbb58670-kube-api-access-8pgff\") pod \"maas-api-revoke-sub-f75163a5-8hx99\" (UID: \"fd839149-8249-44f4-baf7-9fcedbb58670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"
Oct 30 20:13:52.704006 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.703957    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd839149-8249-44f4-baf7-9fcedbb58670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f75163a5-8hx99\" (UID: \"fd839149-8249-44f4-baf7-9fcedbb58670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"
Oct 30 20:13:52.704464 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.704438    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd839149-8249-44f4-baf7-9fcedbb58670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f75163a5-8hx99\" (UID: \"fd839149-8249-44f4-baf7-9fcedbb58670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"
Oct 30 20:13:52.706622 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.706602    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd839149-8249-44f4-baf7-9fcedbb58670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f75163a5-8hx99\" (UID: \"fd839149-8249-44f4-baf7-9fcedbb58670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"
Oct 30 20:13:52.716280 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.716256    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pgff\" (UniqueName: \"kubernetes.io/projected/fd839149-8249-44f4-baf7-9fcedbb58670-kube-api-access-8pgff\") pod \"maas-api-revoke-sub-f75163a5-8hx99\" (UID: \"fd839149-8249-44f4-baf7-9fcedbb58670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"
Oct 30 20:13:52.817595 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:52.817518    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"
Oct 30 20:13:53.109635 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:53.109605    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"]
Oct 30 20:13:53.110908 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:13:53.110879    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd839149_8249_44f4_baf7_9fcedbb58670.slice/crio-9f61fd776f4f39db4d383127bbf65d93d3d861801263446f25717c5ec92bb731 WatchSource:0}: Error finding container 9f61fd776f4f39db4d383127bbf65d93d3d861801263446f25717c5ec92bb731: Status 404 returned error can't find the container with id 9f61fd776f4f39db4d383127bbf65d93d3d861801263446f25717c5ec92bb731
Oct 30 20:13:53.721082 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:53.721000    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99" event={"ID":"fd839149-8249-44f4-baf7-9fcedbb58670","Type":"ContainerStarted","Data":"8bd35d6343652e88c6d0b90c5d3ce84e77d73badd70f2b233f50597bd1e3744d"}
Oct 30 20:13:53.721082 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:53.721061    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99" event={"ID":"fd839149-8249-44f4-baf7-9fcedbb58670","Type":"ContainerStarted","Data":"9f61fd776f4f39db4d383127bbf65d93d3d861801263446f25717c5ec92bb731"}
Oct 30 20:13:53.744767 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:53.744717    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99" podStartSLOduration=1.744702781 podStartE2EDuration="1.744702781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:53.743679468 +0000 UTC m=+2633.636293797" watchObservedRunningTime="2026-10-30 20:13:53.744702781 +0000 UTC m=+2633.637317105"
Oct 30 20:13:54.729528 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:54.729491    4823 generic.go:356] "Generic (PLEG): container finished" podID="fd839149-8249-44f4-baf7-9fcedbb58670" containerID="8bd35d6343652e88c6d0b90c5d3ce84e77d73badd70f2b233f50597bd1e3744d" exitCode=0
Oct 30 20:13:54.729958 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:54.729570    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99" event={"ID":"fd839149-8249-44f4-baf7-9fcedbb58670","Type":"ContainerDied","Data":"8bd35d6343652e88c6d0b90c5d3ce84e77d73badd70f2b233f50597bd1e3744d"}
Oct 30 20:13:55.903620 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:55.903590    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"
Oct 30 20:13:56.042799 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:56.042718    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd839149-8249-44f4-baf7-9fcedbb58670-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd839149-8249-44f4-baf7-9fcedbb58670-maas-api-openshift-ca\") pod \"fd839149-8249-44f4-baf7-9fcedbb58670\" (UID: \"fd839149-8249-44f4-baf7-9fcedbb58670\") "
Oct 30 20:13:56.042956 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:56.042805    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd839149-8249-44f4-baf7-9fcedbb58670-kube-api-access-8pgff\" (UniqueName: \"kubernetes.io/projected/fd839149-8249-44f4-baf7-9fcedbb58670-kube-api-access-8pgff\") pod \"fd839149-8249-44f4-baf7-9fcedbb58670\" (UID: \"fd839149-8249-44f4-baf7-9fcedbb58670\") "
Oct 30 20:13:56.042956 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:56.042905    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd839149-8249-44f4-baf7-9fcedbb58670-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd839149-8249-44f4-baf7-9fcedbb58670-maas-api-xks-ca\") pod \"fd839149-8249-44f4-baf7-9fcedbb58670\" (UID: \"fd839149-8249-44f4-baf7-9fcedbb58670\") "
Oct 30 20:13:56.043136 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:56.043114    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd839149-8249-44f4-baf7-9fcedbb58670-maas-api-openshift-ca" pod "fd839149-8249-44f4-baf7-9fcedbb58670" (UID: "fd839149-8249-44f4-baf7-9fcedbb58670"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:56.043300 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:56.043272    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd839149-8249-44f4-baf7-9fcedbb58670-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:56.044957 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:56.044937    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd839149-8249-44f4-baf7-9fcedbb58670-maas-api-xks-ca" pod "fd839149-8249-44f4-baf7-9fcedbb58670" (UID: "fd839149-8249-44f4-baf7-9fcedbb58670"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:56.045095 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:56.045075    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd839149-8249-44f4-baf7-9fcedbb58670-kube-api-access-8pgff" pod "fd839149-8249-44f4-baf7-9fcedbb58670" (UID: "fd839149-8249-44f4-baf7-9fcedbb58670"). InnerVolumeSpecName "kube-api-access-8pgff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:56.144194 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:56.144158    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pgff\" (UniqueName: \"kubernetes.io/projected/fd839149-8249-44f4-baf7-9fcedbb58670-kube-api-access-8pgff\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:56.144194 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:56.144189    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd839149-8249-44f4-baf7-9fcedbb58670-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:56.630122 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:56.630091    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:13:56.746097 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:56.746071    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"
Oct 30 20:13:56.746270 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:56.746106    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99" event={"ID":"fd839149-8249-44f4-baf7-9fcedbb58670","Type":"ContainerDied","Data":"9f61fd776f4f39db4d383127bbf65d93d3d861801263446f25717c5ec92bb731"}
Oct 30 20:13:56.746270 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:56.746161    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f61fd776f4f39db4d383127bbf65d93d3d861801263446f25717c5ec92bb731"
Oct 30 20:13:58.846113 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:58.846066    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main" containerID="cri-o://b0ebe5b25ec0ec2a4cd09f564bf2f443dadce7394d195e46df816835ff7154d0" gracePeriod=45
Oct 30 20:13:58.883567 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:58.883525    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main" containerID="cri-o://10974dbd2acbb0f904f1bf2f5b69e10228b7e498a7681377c119168bc30e4277" gracePeriod=45
Oct 30 20:13:59.202819 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.202788    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 20:13:59.235611 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.235575    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 20:13:59.290567 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.290524    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-dshm\") pod \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") "
Oct 30 20:13:59.290773 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.290581    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795d618e-c7a6-4495-8784-9e5c8e1b00f0-kube-api-access-mncgt\" (UniqueName: \"kubernetes.io/projected/795d618e-c7a6-4495-8784-9e5c8e1b00f0-kube-api-access-mncgt\") pod \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") "
Oct 30 20:13:59.290773 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.290681    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-home\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-home\") pod \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") "
Oct 30 20:13:59.290773 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.290765    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-tmp-dir\") pod \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") "
Oct 30 20:13:59.291013 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.290801    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/795d618e-c7a6-4495-8784-9e5c8e1b00f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/795d618e-c7a6-4495-8784-9e5c8e1b00f0-tls-certs\") pod \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") "
Oct 30 20:13:59.291013 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.290996    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-model-cache\") pod \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\" (UID: \"795d618e-c7a6-4495-8784-9e5c8e1b00f0\") "
Oct 30 20:13:59.291745 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.291123    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-home" pod "795d618e-c7a6-4495-8784-9e5c8e1b00f0" (UID: "795d618e-c7a6-4495-8784-9e5c8e1b00f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:59.291745 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.291310    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-model-cache" pod "795d618e-c7a6-4495-8784-9e5c8e1b00f0" (UID: "795d618e-c7a6-4495-8784-9e5c8e1b00f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:59.291745 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.291560    4823 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-model-cache\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.291745 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.291578    4823 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-home\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.291745 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.291705    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-tmp-dir" pod "795d618e-c7a6-4495-8784-9e5c8e1b00f0" (UID: "795d618e-c7a6-4495-8784-9e5c8e1b00f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:59.294283 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.294240    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-dshm" pod "795d618e-c7a6-4495-8784-9e5c8e1b00f0" (UID: "795d618e-c7a6-4495-8784-9e5c8e1b00f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:59.294404 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.294377    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795d618e-c7a6-4495-8784-9e5c8e1b00f0-tls-certs" pod "795d618e-c7a6-4495-8784-9e5c8e1b00f0" (UID: "795d618e-c7a6-4495-8784-9e5c8e1b00f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:59.294599 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.294573    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795d618e-c7a6-4495-8784-9e5c8e1b00f0-kube-api-access-mncgt" pod "795d618e-c7a6-4495-8784-9e5c8e1b00f0" (UID: "795d618e-c7a6-4495-8784-9e5c8e1b00f0"). InnerVolumeSpecName "kube-api-access-mncgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:59.392820 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.392737    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-home\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-home\") pod \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") "
Oct 30 20:13:59.393007 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.392837    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-kube-api-access-svqnf\" (UniqueName: \"kubernetes.io/projected/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-kube-api-access-svqnf\") pod \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") "
Oct 30 20:13:59.393007 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.392953    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-dshm\") pod \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") "
Oct 30 20:13:59.393007 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.392982    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-tls-certs\") pod \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") "
Oct 30 20:13:59.393121 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.393015    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-tmp-dir\") pod \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") "
Oct 30 20:13:59.393121 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.393063    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-model-cache\") pod \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\" (UID: \"6a2ff89f-2dce-4e68-a8b8-443581fb78ee\") "
Oct 30 20:13:59.393121 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.393070    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-home" pod "6a2ff89f-2dce-4e68-a8b8-443581fb78ee" (UID: "6a2ff89f-2dce-4e68-a8b8-443581fb78ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:59.393374 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.393308    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-tmp-dir" pod "6a2ff89f-2dce-4e68-a8b8-443581fb78ee" (UID: "6a2ff89f-2dce-4e68-a8b8-443581fb78ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:59.393515 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.393377    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-model-cache" pod "6a2ff89f-2dce-4e68-a8b8-443581fb78ee" (UID: "6a2ff89f-2dce-4e68-a8b8-443581fb78ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:59.393729 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.393710    4823 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-home\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.393821 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.393733    4823 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-dshm\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.393821 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.393747    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mncgt\" (UniqueName: \"kubernetes.io/projected/795d618e-c7a6-4495-8784-9e5c8e1b00f0-kube-api-access-mncgt\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.393821 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.393763    4823 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-tmp-dir\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.393821 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.393777    4823 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-model-cache\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.393821 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.393790    4823 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/795d618e-c7a6-4495-8784-9e5c8e1b00f0-tmp-dir\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.393821 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.393803    4823 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/795d618e-c7a6-4495-8784-9e5c8e1b00f0-tls-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.395135 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.395115    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-dshm" pod "6a2ff89f-2dce-4e68-a8b8-443581fb78ee" (UID: "6a2ff89f-2dce-4e68-a8b8-443581fb78ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:59.395615 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.395596    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-tls-certs" pod "6a2ff89f-2dce-4e68-a8b8-443581fb78ee" (UID: "6a2ff89f-2dce-4e68-a8b8-443581fb78ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:59.395674 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.395622    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-kube-api-access-svqnf" pod "6a2ff89f-2dce-4e68-a8b8-443581fb78ee" (UID: "6a2ff89f-2dce-4e68-a8b8-443581fb78ee"). InnerVolumeSpecName "kube-api-access-svqnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:59.494533 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.494493    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svqnf\" (UniqueName: \"kubernetes.io/projected/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-kube-api-access-svqnf\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.494533 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.494526    4823 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-dshm\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.494533 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.494540    4823 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a2ff89f-2dce-4e68-a8b8-443581fb78ee-tls-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:59.772035 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.771944    4823 generic.go:356] "Generic (PLEG): container finished" podID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerID="10974dbd2acbb0f904f1bf2f5b69e10228b7e498a7681377c119168bc30e4277" exitCode=0
Oct 30 20:13:59.772035 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.771994    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"
Oct 30 20:13:59.772292 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.772033    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" event={"ID":"6a2ff89f-2dce-4e68-a8b8-443581fb78ee","Type":"ContainerDied","Data":"10974dbd2acbb0f904f1bf2f5b69e10228b7e498a7681377c119168bc30e4277"}
Oct 30 20:13:59.772292 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.772085    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk" event={"ID":"6a2ff89f-2dce-4e68-a8b8-443581fb78ee","Type":"ContainerDied","Data":"5f78420c1333735e064ec3ce3461ef6f70bf1b9b9a2e7311098a113475e123d7"}
Oct 30 20:13:59.772292 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.772110    4823 scope.go:122] "RemoveContainer" containerID="10974dbd2acbb0f904f1bf2f5b69e10228b7e498a7681377c119168bc30e4277"
Oct 30 20:13:59.773588 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.773560    4823 generic.go:356] "Generic (PLEG): container finished" podID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerID="b0ebe5b25ec0ec2a4cd09f564bf2f443dadce7394d195e46df816835ff7154d0" exitCode=0
Oct 30 20:13:59.773710 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.773595    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"
Oct 30 20:13:59.773710 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.773623    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" event={"ID":"795d618e-c7a6-4495-8784-9e5c8e1b00f0","Type":"ContainerDied","Data":"b0ebe5b25ec0ec2a4cd09f564bf2f443dadce7394d195e46df816835ff7154d0"}
Oct 30 20:13:59.773710 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.773671    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz" event={"ID":"795d618e-c7a6-4495-8784-9e5c8e1b00f0","Type":"ContainerDied","Data":"b99bc0bbe01816aa8b688452dbb5a8b74cf3aa8d5315052b825e2e4f07f5f625"}
Oct 30 20:13:59.785458 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.785279    4823 scope.go:122] "RemoveContainer" containerID="10974dbd2acbb0f904f1bf2f5b69e10228b7e498a7681377c119168bc30e4277"
Oct 30 20:13:59.785724 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:13:59.785697    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10974dbd2acbb0f904f1bf2f5b69e10228b7e498a7681377c119168bc30e4277\": container with ID starting with 10974dbd2acbb0f904f1bf2f5b69e10228b7e498a7681377c119168bc30e4277 not found: ID does not exist" containerID="10974dbd2acbb0f904f1bf2f5b69e10228b7e498a7681377c119168bc30e4277"
Oct 30 20:13:59.785788 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.785731    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10974dbd2acbb0f904f1bf2f5b69e10228b7e498a7681377c119168bc30e4277"} err="failed to get container status \"10974dbd2acbb0f904f1bf2f5b69e10228b7e498a7681377c119168bc30e4277\": rpc error: code = NotFound desc = could not find container \"10974dbd2acbb0f904f1bf2f5b69e10228b7e498a7681377c119168bc30e4277\": container with ID starting with 10974dbd2acbb0f904f1bf2f5b69e10228b7e498a7681377c119168bc30e4277 not found: ID does not exist"
Oct 30 20:13:59.785788 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.785748    4823 scope.go:122] "RemoveContainer" containerID="b0ebe5b25ec0ec2a4cd09f564bf2f443dadce7394d195e46df816835ff7154d0"
Oct 30 20:13:59.797679 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.797267    4823 scope.go:122] "RemoveContainer" containerID="b0ebe5b25ec0ec2a4cd09f564bf2f443dadce7394d195e46df816835ff7154d0"
Oct 30 20:13:59.797679 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:13:59.797573    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ebe5b25ec0ec2a4cd09f564bf2f443dadce7394d195e46df816835ff7154d0\": container with ID starting with b0ebe5b25ec0ec2a4cd09f564bf2f443dadce7394d195e46df816835ff7154d0 not found: ID does not exist" containerID="b0ebe5b25ec0ec2a4cd09f564bf2f443dadce7394d195e46df816835ff7154d0"
Oct 30 20:13:59.797679 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.797605    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ebe5b25ec0ec2a4cd09f564bf2f443dadce7394d195e46df816835ff7154d0"} err="failed to get container status \"b0ebe5b25ec0ec2a4cd09f564bf2f443dadce7394d195e46df816835ff7154d0\": rpc error: code = NotFound desc = could not find container \"b0ebe5b25ec0ec2a4cd09f564bf2f443dadce7394d195e46df816835ff7154d0\": container with ID starting with b0ebe5b25ec0ec2a4cd09f564bf2f443dadce7394d195e46df816835ff7154d0 not found: ID does not exist"
Oct 30 20:13:59.801728 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.801658    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"]
Oct 30 20:13:59.806630 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.806607    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/premium-w0-ced832-kserve-59d489cd49-7xmmk"]
Oct 30 20:13:59.813561 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.813538    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"]
Oct 30 20:13:59.822016 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:13:59.821995    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ced832/distinct-2-w0-ced832-kserve-6557d8cf7f-gmmgz"]
Oct 30 20:14:00.281437 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:00.281406    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" path="/var/lib/kubelet/pods/6a2ff89f-2dce-4e68-a8b8-443581fb78ee/volumes"
Oct 30 20:14:00.281894 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:00.281878    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" path="/var/lib/kubelet/pods/795d618e-c7a6-4495-8784-9e5c8e1b00f0/volumes"
Oct 30 20:14:02.013763 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:14:02.013716    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:14:07.534786 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:07.534749    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:14:17.557701 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:17.557623    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"]
Oct 30 20:14:17.558126 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:17.557903    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44" podUID="dd984f9d-dcce-452e-a843-e07894b05c69" containerName="istio-proxy" containerID="cri-o://0da710fce960157285e7ccb5eb1697e859a7c1f31b0cb06eb0c9ab956c20719e" gracePeriod=30
Oct 30 20:14:18.317387 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:18.317337    4823 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:14:18.317564 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:18.317430    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44" podUID="dd984f9d-dcce-452e-a843-e07894b05c69" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:14:22.831722 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.831654    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 20:14:22.896101 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.896070    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-workload-socket\") pod \"dd984f9d-dcce-452e-a843-e07894b05c69\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") "
Oct 30 20:14:22.896304 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.896111    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-istio-data\") pod \"dd984f9d-dcce-452e-a843-e07894b05c69\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") "
Oct 30 20:14:22.896304 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.896206    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd984f9d-dcce-452e-a843-e07894b05c69-istio-token\" (UniqueName: \"kubernetes.io/projected/dd984f9d-dcce-452e-a843-e07894b05c69-istio-token\") pod \"dd984f9d-dcce-452e-a843-e07894b05c69\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") "
Oct 30 20:14:22.896448 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.896300    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-credential-socket\") pod \"dd984f9d-dcce-452e-a843-e07894b05c69\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") "
Oct 30 20:14:22.896448 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.896343    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-istio-envoy\") pod \"dd984f9d-dcce-452e-a843-e07894b05c69\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") "
Oct 30 20:14:22.896448 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.896398    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd984f9d-dcce-452e-a843-e07894b05c69-kube-api-access-df4gh\" (UniqueName: \"kubernetes.io/projected/dd984f9d-dcce-452e-a843-e07894b05c69-kube-api-access-df4gh\") pod \"dd984f9d-dcce-452e-a843-e07894b05c69\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") "
Oct 30 20:14:22.896448 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.896399    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-workload-socket" pod "dd984f9d-dcce-452e-a843-e07894b05c69" (UID: "dd984f9d-dcce-452e-a843-e07894b05c69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:22.896666 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.896456    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-istio-data" pod "dd984f9d-dcce-452e-a843-e07894b05c69" (UID: "dd984f9d-dcce-452e-a843-e07894b05c69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:22.896666 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.896463    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-workload-certs\") pod \"dd984f9d-dcce-452e-a843-e07894b05c69\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") "
Oct 30 20:14:22.896666 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.896548    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd984f9d-dcce-452e-a843-e07894b05c69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd984f9d-dcce-452e-a843-e07894b05c69-istio-podinfo\") pod \"dd984f9d-dcce-452e-a843-e07894b05c69\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") "
Oct 30 20:14:22.896666 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.896637    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-workload-certs" pod "dd984f9d-dcce-452e-a843-e07894b05c69" (UID: "dd984f9d-dcce-452e-a843-e07894b05c69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:22.896887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.896670    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd984f9d-dcce-452e-a843-e07894b05c69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd984f9d-dcce-452e-a843-e07894b05c69-istiod-ca-cert\") pod \"dd984f9d-dcce-452e-a843-e07894b05c69\" (UID: \"dd984f9d-dcce-452e-a843-e07894b05c69\") "
Oct 30 20:14:22.896887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.896819    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-credential-socket" pod "dd984f9d-dcce-452e-a843-e07894b05c69" (UID: "dd984f9d-dcce-452e-a843-e07894b05c69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:22.897197 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.897175    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd984f9d-dcce-452e-a843-e07894b05c69-istiod-ca-cert" pod "dd984f9d-dcce-452e-a843-e07894b05c69" (UID: "dd984f9d-dcce-452e-a843-e07894b05c69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:22.897335 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.897272    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:22.897457 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.897443    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:22.897554 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.897543    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:22.897646 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.897634    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:22.899300 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.899270    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd984f9d-dcce-452e-a843-e07894b05c69-kube-api-access-df4gh" pod "dd984f9d-dcce-452e-a843-e07894b05c69" (UID: "dd984f9d-dcce-452e-a843-e07894b05c69"). InnerVolumeSpecName "kube-api-access-df4gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:22.899458 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.899434    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-istio-envoy" pod "dd984f9d-dcce-452e-a843-e07894b05c69" (UID: "dd984f9d-dcce-452e-a843-e07894b05c69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:14:22.899627 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.899607    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd984f9d-dcce-452e-a843-e07894b05c69-istio-token" pod "dd984f9d-dcce-452e-a843-e07894b05c69" (UID: "dd984f9d-dcce-452e-a843-e07894b05c69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:22.899711 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.899691    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd984f9d-dcce-452e-a843-e07894b05c69-istio-podinfo" pod "dd984f9d-dcce-452e-a843-e07894b05c69" (UID: "dd984f9d-dcce-452e-a843-e07894b05c69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:14:22.958187 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.958155    4823 generic.go:356] "Generic (PLEG): container finished" podID="dd984f9d-dcce-452e-a843-e07894b05c69" containerID="0da710fce960157285e7ccb5eb1697e859a7c1f31b0cb06eb0c9ab956c20719e" exitCode=0
Oct 30 20:14:22.958359 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.958207    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"
Oct 30 20:14:22.958359 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.958234    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44" event={"ID":"dd984f9d-dcce-452e-a843-e07894b05c69","Type":"ContainerDied","Data":"0da710fce960157285e7ccb5eb1697e859a7c1f31b0cb06eb0c9ab956c20719e"}
Oct 30 20:14:22.958359 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.958302    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44" event={"ID":"dd984f9d-dcce-452e-a843-e07894b05c69","Type":"ContainerDied","Data":"9815397033e9a6077131dc6568140947a84716f0b5113ad0bb0553a53fd546f0"}
Oct 30 20:14:22.958359 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.958331    4823 scope.go:122] "RemoveContainer" containerID="0da710fce960157285e7ccb5eb1697e859a7c1f31b0cb06eb0c9ab956c20719e"
Oct 30 20:14:22.969652 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.969633    4823 scope.go:122] "RemoveContainer" containerID="91ce26b56cb6dc68cd1b497564dafeac233a8113118979450423f525660b5e3c"
Oct 30 20:14:22.979205 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.979187    4823 scope.go:122] "RemoveContainer" containerID="0da710fce960157285e7ccb5eb1697e859a7c1f31b0cb06eb0c9ab956c20719e"
Oct 30 20:14:22.979485 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:14:22.979466    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da710fce960157285e7ccb5eb1697e859a7c1f31b0cb06eb0c9ab956c20719e\": container with ID starting with 0da710fce960157285e7ccb5eb1697e859a7c1f31b0cb06eb0c9ab956c20719e not found: ID does not exist" containerID="0da710fce960157285e7ccb5eb1697e859a7c1f31b0cb06eb0c9ab956c20719e"
Oct 30 20:14:22.979532 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.979493    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da710fce960157285e7ccb5eb1697e859a7c1f31b0cb06eb0c9ab956c20719e"} err="failed to get container status \"0da710fce960157285e7ccb5eb1697e859a7c1f31b0cb06eb0c9ab956c20719e\": rpc error: code = NotFound desc = could not find container \"0da710fce960157285e7ccb5eb1697e859a7c1f31b0cb06eb0c9ab956c20719e\": container with ID starting with 0da710fce960157285e7ccb5eb1697e859a7c1f31b0cb06eb0c9ab956c20719e not found: ID does not exist"
Oct 30 20:14:22.979532 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.979510    4823 scope.go:122] "RemoveContainer" containerID="91ce26b56cb6dc68cd1b497564dafeac233a8113118979450423f525660b5e3c"
Oct 30 20:14:22.979781 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:14:22.979763    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ce26b56cb6dc68cd1b497564dafeac233a8113118979450423f525660b5e3c\": container with ID starting with 91ce26b56cb6dc68cd1b497564dafeac233a8113118979450423f525660b5e3c not found: ID does not exist" containerID="91ce26b56cb6dc68cd1b497564dafeac233a8113118979450423f525660b5e3c"
Oct 30 20:14:22.979836 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.979790    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ce26b56cb6dc68cd1b497564dafeac233a8113118979450423f525660b5e3c"} err="failed to get container status \"91ce26b56cb6dc68cd1b497564dafeac233a8113118979450423f525660b5e3c\": rpc error: code = NotFound desc = could not find container \"91ce26b56cb6dc68cd1b497564dafeac233a8113118979450423f525660b5e3c\": container with ID starting with 91ce26b56cb6dc68cd1b497564dafeac233a8113118979450423f525660b5e3c not found: ID does not exist"
Oct 30 20:14:22.991375 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.991354    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"]
Oct 30 20:14:22.994328 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.994306    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-57p44"]
Oct 30 20:14:22.998835 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.998809    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd984f9d-dcce-452e-a843-e07894b05c69-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:22.998835 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.998831    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd984f9d-dcce-452e-a843-e07894b05c69-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:22.998981 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.998845    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df4gh\" (UniqueName: \"kubernetes.io/projected/dd984f9d-dcce-452e-a843-e07894b05c69-kube-api-access-df4gh\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:22.998981 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.998888    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd984f9d-dcce-452e-a843-e07894b05c69-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:22.998981 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:22.998902    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd984f9d-dcce-452e-a843-e07894b05c69-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:24.279284 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:24.279244    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd984f9d-dcce-452e-a843-e07894b05c69" path="/var/lib/kubelet/pods/dd984f9d-dcce-452e-a843-e07894b05c69/volumes"
Oct 30 20:14:24.712534 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:24.712499    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:14:32.015074 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:14:32.015033    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:14:32.126718 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.126681    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"]
Oct 30 20:14:32.127498 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.127475    4823 cpu_manager.go:414] "containerMap: removing container" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main"
Oct 30 20:14:32.127558 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.127503    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main"
Oct 30 20:14:32.127558 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.127533    4823 cpu_manager.go:414] "containerMap: removing container" podUID="dd984f9d-dcce-452e-a843-e07894b05c69" containerName="istio-proxy"
Oct 30 20:14:32.127558 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.127541    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd984f9d-dcce-452e-a843-e07894b05c69" containerName="istio-proxy"
Oct 30 20:14:32.127558 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.127552    4823 cpu_manager.go:414] "containerMap: removing container" podUID="fd839149-8249-44f4-baf7-9fcedbb58670" containerName="revoke-keys"
Oct 30 20:14:32.127694 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.127561    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd839149-8249-44f4-baf7-9fcedbb58670" containerName="revoke-keys"
Oct 30 20:14:32.127694 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.127604    4823 cpu_manager.go:414] "containerMap: removing container" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main"
Oct 30 20:14:32.127694 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.127611    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main"
Oct 30 20:14:32.127816 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.127745    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd984f9d-dcce-452e-a843-e07894b05c69" containerName="istio-proxy"
Oct 30 20:14:32.127816 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.127755    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd839149-8249-44f4-baf7-9fcedbb58670" containerName="revoke-keys"
Oct 30 20:14:32.127816 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.127772    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee23fe73-0972-42d0-b734-f4c3748354f9" containerName="istio-proxy"
Oct 30 20:14:32.127816 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.127784    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="795d618e-c7a6-4495-8784-9e5c8e1b00f0" containerName="main"
Oct 30 20:14:32.127816 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.127795    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd984f9d-dcce-452e-a843-e07894b05c69" containerName="istio-proxy"
Oct 30 20:14:32.127816 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.127807    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a2ff89f-2dce-4e68-a8b8-443581fb78ee" containerName="main"
Oct 30 20:14:32.132075 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.132044    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.154411 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.154378    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"]
Oct 30 20:14:32.304016 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.303933    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.304016 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.303985    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.304211 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.304091    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.304211 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.304121    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.304291 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.304240    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.304291 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.304277    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.304355 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.304342    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d69555e5-864f-459f-bb32-a8c4fd803b51-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.304395 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.304366    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.304395 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.304382    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkcbn\" (UniqueName: \"kubernetes.io/projected/d69555e5-864f-459f-bb32-a8c4fd803b51-kube-api-access-vkcbn\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.405537 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.405495    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.405723 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.405550    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.405723 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.405613    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.405723 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.405653    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.405936 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.405737    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.405936 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.405792    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.405936 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.405849    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d69555e5-864f-459f-bb32-a8c4fd803b51-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.405936 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.405899    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.405936 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.405926    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkcbn\" (UniqueName: \"kubernetes.io/projected/d69555e5-864f-459f-bb32-a8c4fd803b51-kube-api-access-vkcbn\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.406284 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.406032    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.406284 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.406139    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.406284 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.406256    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.406543 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.406517    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.406839 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.406816    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d69555e5-864f-459f-bb32-a8c4fd803b51-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.408321 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.408300    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.408655 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.408635    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.422531 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.422504    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.423183 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.423165    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkcbn\" (UniqueName: \"kubernetes.io/projected/d69555e5-864f-459f-bb32-a8c4fd803b51-kube-api-access-vkcbn\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.470294 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.470266    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:32.760508 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.760478    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"]
Oct 30 20:14:32.762124 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:14:32.762094    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69555e5_864f_459f_bb32_a8c4fd803b51.slice/crio-f087cbe92bc00050bb10a21ceb1f59da71f70e54c671ffb3854e858ba34f2518 WatchSource:0}: Error finding container f087cbe92bc00050bb10a21ceb1f59da71f70e54c671ffb3854e858ba34f2518: Status 404 returned error can't find the container with id f087cbe92bc00050bb10a21ceb1f59da71f70e54c671ffb3854e858ba34f2518
Oct 30 20:14:32.764637 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.764602    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:14:32.764709 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.764667    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:14:32.764709 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:32.764696    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:14:33.043482 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:33.043402    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth" event={"ID":"d69555e5-864f-459f-bb32-a8c4fd803b51","Type":"ContainerStarted","Data":"e565066f28ab9679fe38b0742af87fca841df578e1624afcc398652226c381de"}
Oct 30 20:14:33.043482 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:33.043453    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth" event={"ID":"d69555e5-864f-459f-bb32-a8c4fd803b51","Type":"ContainerStarted","Data":"f087cbe92bc00050bb10a21ceb1f59da71f70e54c671ffb3854e858ba34f2518"}
Oct 30 20:14:33.065546 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:33.065486    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth" podStartSLOduration=1.065469429 podStartE2EDuration="1.065469429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:33.064927339 +0000 UTC m=+2672.957541657" watchObservedRunningTime="2026-10-30 20:14:33.065469429 +0000 UTC m=+2672.958083754"
Oct 30 20:14:33.477015 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:33.476974    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:33.477795 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:33.477763    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 20:14:33.477988 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:33.477835    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth" podUID="d69555e5-864f-459f-bb32-a8c4fd803b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 20:14:33.525495 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:33.525462    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:14:34.473893 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:34.473226    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 20:14:34.473893 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:34.473307    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth" podUID="d69555e5-864f-459f-bb32-a8c4fd803b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 20:14:35.475887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:35.475299    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 20:14:35.475887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:35.475406    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth" podUID="d69555e5-864f-459f-bb32-a8c4fd803b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 20:14:36.477764 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:36.477278    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 20:14:36.477764 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:36.477342    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth" podUID="d69555e5-864f-459f-bb32-a8c4fd803b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 20:14:37.476914 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:37.474318    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 20:14:37.476914 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:37.474383    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth" podUID="d69555e5-864f-459f-bb32-a8c4fd803b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 20:14:38.472887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:38.471247    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 20:14:38.472887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:38.471313    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth" podUID="d69555e5-864f-459f-bb32-a8c4fd803b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 20:14:39.678016 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:39.677982    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:39.679170 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:39.679134    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:39.680384 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:39.680358    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:14:47.155846 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.155800    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"]
Oct 30 20:14:47.157100 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.157073    4823 cpu_manager.go:414] "containerMap: removing container" podUID="dd984f9d-dcce-452e-a843-e07894b05c69" containerName="istio-proxy"
Oct 30 20:14:47.157100 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.157099    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd984f9d-dcce-452e-a843-e07894b05c69" containerName="istio-proxy"
Oct 30 20:14:47.163578 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.163553    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.176809 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.176773    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"]
Oct 30 20:14:47.203171 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.203129    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.203171 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.203163    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.203454 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.203189    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.203454 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.203315    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.203454 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.203361    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.203454 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.203392    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gsbs\" (UniqueName: \"kubernetes.io/projected/251fadc1-f1f5-431d-9286-5bd16b99733e-kube-api-access-8gsbs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.203699 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.203461    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.203699 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.203557    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.203874 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.203832    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251fadc1-f1f5-431d-9286-5bd16b99733e-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.305681 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.305646    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251fadc1-f1f5-431d-9286-5bd16b99733e-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.305911 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.305787    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.305911 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.305806    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.305911 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.305831    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.305911 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.305850    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.305911 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.305909    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.306227 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.305940    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gsbs\" (UniqueName: \"kubernetes.io/projected/251fadc1-f1f5-431d-9286-5bd16b99733e-kube-api-access-8gsbs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.306227 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.306012    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.306227 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.306041    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.306395 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.306341    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.306458 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.306391    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.306519 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.306496    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.306624 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.306600    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251fadc1-f1f5-431d-9286-5bd16b99733e-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.306680 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.306626    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.308331 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.308307    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.308596 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.308578    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.321780 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.321753    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.322123 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.322099    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gsbs\" (UniqueName: \"kubernetes.io/projected/251fadc1-f1f5-431d-9286-5bd16b99733e-kube-api-access-8gsbs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.482956 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.482847    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:47.814343 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.814318    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"]
Oct 30 20:14:47.815749 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:14:47.815721    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251fadc1_f1f5_431d_9286_5bd16b99733e.slice/crio-f49ac953c337a4b7a3fc3184a8ae40d46236a82accf065df862661f0969fe05e WatchSource:0}: Error finding container f49ac953c337a4b7a3fc3184a8ae40d46236a82accf065df862661f0969fe05e: Status 404 returned error can't find the container with id f49ac953c337a4b7a3fc3184a8ae40d46236a82accf065df862661f0969fe05e
Oct 30 20:14:47.818039 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.818012    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:14:47.818120 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.818093    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:14:47.818170 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:47.818126    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:14:48.191493 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:48.191454    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv" event={"ID":"251fadc1-f1f5-431d-9286-5bd16b99733e","Type":"ContainerStarted","Data":"b8476dea9d1a0ee063810078624f0450a4f0b4a58a5db46c8f7cb63bcc3bb8e8"}
Oct 30 20:14:48.191965 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:48.191512    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv" event={"ID":"251fadc1-f1f5-431d-9286-5bd16b99733e","Type":"ContainerStarted","Data":"f49ac953c337a4b7a3fc3184a8ae40d46236a82accf065df862661f0969fe05e"}
Oct 30 20:14:48.236674 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:48.236609    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv" podStartSLOduration=1.23658752 podStartE2EDuration="1.23658752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:48.234185237 +0000 UTC m=+2688.126799561" watchObservedRunningTime="2026-10-30 20:14:48.23658752 +0000 UTC m=+2688.129201845"
Oct 30 20:14:48.483372 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:48.483270    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:48.488421 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:48.488149    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 20:14:48.488421 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:48.488218    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv" podUID="251fadc1-f1f5-431d-9286-5bd16b99733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 20:14:48.806502 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:48.806427    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:14:49.483918 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:49.483847    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 20:14:49.484346 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:49.483931    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv" podUID="251fadc1-f1f5-431d-9286-5bd16b99733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 20:14:50.483490 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:50.483447    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 20:14:50.483766 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:50.483515    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv" podUID="251fadc1-f1f5-431d-9286-5bd16b99733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 20:14:51.483437 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:51.483391    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 20:14:51.483851 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:51.483461    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv" podUID="251fadc1-f1f5-431d-9286-5bd16b99733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 20:14:52.483364 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:52.483325    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 20:14:52.483640 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:52.483382    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv" podUID="251fadc1-f1f5-431d-9286-5bd16b99733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 20:14:53.483338 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:53.483291    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 20:14:53.483526 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:53.483362    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv" podUID="251fadc1-f1f5-431d-9286-5bd16b99733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 20:14:54.487601 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:54.487570    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:54.488019 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:54.488001    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:54.488624 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:54.488605    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:14:58.530344 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:58.530304    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"]
Oct 30 20:14:59.647771 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:14:59.647732    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:15:00.208815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.208774    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x"]
Oct 30 20:15:00.214441 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.214405    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x"
Oct 30 20:15:00.229336 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.229150    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:15:00.229336 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.229302    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:15:00.240854 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.237552    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:15:00.240854 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.238089    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph"]
Oct 30 20:15:00.293887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.279387    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x"]
Oct 30 20:15:00.341508 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.339930    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph"
Oct 30 20:15:00.424178 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.424141    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee26e2d8-c730-4c60-97ce-f76495b990a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x\" (UID: \"ee26e2d8-c730-4c60-97ce-f76495b990a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x"
Oct 30 20:15:00.453801 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.441617    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph"]
Oct 30 20:15:00.453801 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.451778    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vx7t\" (UniqueName: \"kubernetes.io/projected/ee26e2d8-c730-4c60-97ce-f76495b990a8-kube-api-access-9vx7t\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x\" (UID: \"ee26e2d8-c730-4c60-97ce-f76495b990a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x"
Oct 30 20:15:00.453801 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.453321    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn9lz\" (UniqueName: \"kubernetes.io/projected/b3a911c9-4eda-46bd-a390-76b95d5e2c60-kube-api-access-vn9lz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph\" (UID: \"b3a911c9-4eda-46bd-a390-76b95d5e2c60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph"
Oct 30 20:15:00.453801 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.453407    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a911c9-4eda-46bd-a390-76b95d5e2c60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph\" (UID: \"b3a911c9-4eda-46bd-a390-76b95d5e2c60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph"
Oct 30 20:15:00.554793 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.554704    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn9lz\" (UniqueName: \"kubernetes.io/projected/b3a911c9-4eda-46bd-a390-76b95d5e2c60-kube-api-access-vn9lz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph\" (UID: \"b3a911c9-4eda-46bd-a390-76b95d5e2c60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph"
Oct 30 20:15:00.554793 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.554760    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a911c9-4eda-46bd-a390-76b95d5e2c60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph\" (UID: \"b3a911c9-4eda-46bd-a390-76b95d5e2c60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph"
Oct 30 20:15:00.555063 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.554986    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee26e2d8-c730-4c60-97ce-f76495b990a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x\" (UID: \"ee26e2d8-c730-4c60-97ce-f76495b990a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x"
Oct 30 20:15:00.555063 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.555032    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vx7t\" (UniqueName: \"kubernetes.io/projected/ee26e2d8-c730-4c60-97ce-f76495b990a8-kube-api-access-9vx7t\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x\" (UID: \"ee26e2d8-c730-4c60-97ce-f76495b990a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x"
Oct 30 20:15:00.555522 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.555501    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a911c9-4eda-46bd-a390-76b95d5e2c60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph\" (UID: \"b3a911c9-4eda-46bd-a390-76b95d5e2c60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph"
Oct 30 20:15:00.555594 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.555571    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee26e2d8-c730-4c60-97ce-f76495b990a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x\" (UID: \"ee26e2d8-c730-4c60-97ce-f76495b990a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x"
Oct 30 20:15:00.575801 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.575770    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vx7t\" (UniqueName: \"kubernetes.io/projected/ee26e2d8-c730-4c60-97ce-f76495b990a8-kube-api-access-9vx7t\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x\" (UID: \"ee26e2d8-c730-4c60-97ce-f76495b990a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x"
Oct 30 20:15:00.575953 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.575881    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn9lz\" (UniqueName: \"kubernetes.io/projected/b3a911c9-4eda-46bd-a390-76b95d5e2c60-kube-api-access-vn9lz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph\" (UID: \"b3a911c9-4eda-46bd-a390-76b95d5e2c60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph"
Oct 30 20:15:00.752209 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.752172    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:15:00.760417 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.760395    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph"
Oct 30 20:15:00.866512 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:00.866477    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x"
Oct 30 20:15:01.056192 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:01.056162    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph"]
Oct 30 20:15:01.057061 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:15:01.057036    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a911c9_4eda_46bd_a390_76b95d5e2c60.slice/crio-e85457ac42853099ae441569bfe926ce39e2fd1584cf729356c9cf32e48fe21d WatchSource:0}: Error finding container e85457ac42853099ae441569bfe926ce39e2fd1584cf729356c9cf32e48fe21d: Status 404 returned error can't find the container with id e85457ac42853099ae441569bfe926ce39e2fd1584cf729356c9cf32e48fe21d
Oct 30 20:15:01.163743 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:01.163716    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x"]
Oct 30 20:15:01.166029 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:15:01.165992    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee26e2d8_c730_4c60_97ce_f76495b990a8.slice/crio-0db182fb471b053f6edfe99dd49ab95b0aad219f43d5381f4d201210c9af6f39 WatchSource:0}: Error finding container 0db182fb471b053f6edfe99dd49ab95b0aad219f43d5381f4d201210c9af6f39: Status 404 returned error can't find the container with id 0db182fb471b053f6edfe99dd49ab95b0aad219f43d5381f4d201210c9af6f39
Oct 30 20:15:01.380809 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:01.380767    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph" event={"ID":"b3a911c9-4eda-46bd-a390-76b95d5e2c60","Type":"ContainerStarted","Data":"e85457ac42853099ae441569bfe926ce39e2fd1584cf729356c9cf32e48fe21d"}
Oct 30 20:15:01.383198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:01.383167    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x" event={"ID":"ee26e2d8-c730-4c60-97ce-f76495b990a8","Type":"ContainerStarted","Data":"0db182fb471b053f6edfe99dd49ab95b0aad219f43d5381f4d201210c9af6f39"}
Oct 30 20:15:01.568576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:01.568551    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:15:01.571499 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:01.571476    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:15:02.015421 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:15:02.015377    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:15:02.392356 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:02.392322    4823 generic.go:356] "Generic (PLEG): container finished" podID="b3a911c9-4eda-46bd-a390-76b95d5e2c60" containerID="9d16d574ef151738e80f93aeb5a68a372aa1dd2efd6607e8f1afeaac32403e21" exitCode=60
Oct 30 20:15:02.392554 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:02.392414    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph" event={"ID":"b3a911c9-4eda-46bd-a390-76b95d5e2c60","Type":"ContainerDied","Data":"9d16d574ef151738e80f93aeb5a68a372aa1dd2efd6607e8f1afeaac32403e21"}
Oct 30 20:15:02.392664 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:02.392647    4823 scope.go:122] "RemoveContainer" containerID="9d16d574ef151738e80f93aeb5a68a372aa1dd2efd6607e8f1afeaac32403e21"
Oct 30 20:15:02.394217 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:02.394128    4823 generic.go:356] "Generic (PLEG): container finished" podID="ee26e2d8-c730-4c60-97ce-f76495b990a8" containerID="c434710412e4a50f8a897d974c9f3ef76544b9e0002fe6a53eea6832ef0766f1" exitCode=60
Oct 30 20:15:02.394300 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:02.394213    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x" event={"ID":"ee26e2d8-c730-4c60-97ce-f76495b990a8","Type":"ContainerDied","Data":"c434710412e4a50f8a897d974c9f3ef76544b9e0002fe6a53eea6832ef0766f1"}
Oct 30 20:15:02.394408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:02.394392    4823 scope.go:122] "RemoveContainer" containerID="c434710412e4a50f8a897d974c9f3ef76544b9e0002fe6a53eea6832ef0766f1"
Oct 30 20:15:03.403152 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:03.403065    4823 generic.go:356] "Generic (PLEG): container finished" podID="b3a911c9-4eda-46bd-a390-76b95d5e2c60" containerID="d16f776a23f997a661b719037ad0053b2403e447f7a6371ee471a342ac59bdcb" exitCode=60
Oct 30 20:15:03.403678 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:03.403155    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph" event={"ID":"b3a911c9-4eda-46bd-a390-76b95d5e2c60","Type":"ContainerDied","Data":"d16f776a23f997a661b719037ad0053b2403e447f7a6371ee471a342ac59bdcb"}
Oct 30 20:15:03.403678 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:03.403219    4823 scope.go:122] "RemoveContainer" containerID="9d16d574ef151738e80f93aeb5a68a372aa1dd2efd6607e8f1afeaac32403e21"
Oct 30 20:15:03.403678 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:03.403527    4823 scope.go:122] "RemoveContainer" containerID="d16f776a23f997a661b719037ad0053b2403e447f7a6371ee471a342ac59bdcb"
Oct 30 20:15:03.403908 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:15:03.403833    4823 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph_odh-ai-gateway-infra(b3a911c9-4eda-46bd-a390-76b95d5e2c60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph" podUID="b3a911c9-4eda-46bd-a390-76b95d5e2c60"
Oct 30 20:15:03.405277 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:03.405250    4823 generic.go:356] "Generic (PLEG): container finished" podID="ee26e2d8-c730-4c60-97ce-f76495b990a8" containerID="8f99d7ce1f6439e62fdbd587f7a64097155fb28eaada5e12d5ed03b9755fc79c" exitCode=60
Oct 30 20:15:03.405379 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:03.405339    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x" event={"ID":"ee26e2d8-c730-4c60-97ce-f76495b990a8","Type":"ContainerDied","Data":"8f99d7ce1f6439e62fdbd587f7a64097155fb28eaada5e12d5ed03b9755fc79c"}
Oct 30 20:15:03.405551 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:03.405537    4823 scope.go:122] "RemoveContainer" containerID="8f99d7ce1f6439e62fdbd587f7a64097155fb28eaada5e12d5ed03b9755fc79c"
Oct 30 20:15:03.405781 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:15:03.405763    4823 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x_odh-ai-gateway-infra(ee26e2d8-c730-4c60-97ce-f76495b990a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x" podUID="ee26e2d8-c730-4c60-97ce-f76495b990a8"
Oct 30 20:15:03.416519 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:03.416499    4823 scope.go:122] "RemoveContainer" containerID="c434710412e4a50f8a897d974c9f3ef76544b9e0002fe6a53eea6832ef0766f1"
Oct 30 20:15:04.419385 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:04.419346    4823 scope.go:122] "RemoveContainer" containerID="d16f776a23f997a661b719037ad0053b2403e447f7a6371ee471a342ac59bdcb"
Oct 30 20:15:04.419854 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:15:04.419649    4823 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph_odh-ai-gateway-infra(b3a911c9-4eda-46bd-a390-76b95d5e2c60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph" podUID="b3a911c9-4eda-46bd-a390-76b95d5e2c60"
Oct 30 20:15:04.420921 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:04.420900    4823 scope.go:122] "RemoveContainer" containerID="8f99d7ce1f6439e62fdbd587f7a64097155fb28eaada5e12d5ed03b9755fc79c"
Oct 30 20:15:04.421137 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:15:04.421119    4823 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x_odh-ai-gateway-infra(ee26e2d8-c730-4c60-97ce-f76495b990a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x" podUID="ee26e2d8-c730-4c60-97ce-f76495b990a8"
Oct 30 20:15:08.007461 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:08.007423    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:15:13.143476 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.143443    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x"]
Oct 30 20:15:13.310211 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.310174    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x"
Oct 30 20:15:13.467257 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.467178    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee26e2d8-c730-4c60-97ce-f76495b990a8-kube-api-access-9vx7t\" (UniqueName: \"kubernetes.io/projected/ee26e2d8-c730-4c60-97ce-f76495b990a8-kube-api-access-9vx7t\") pod \"ee26e2d8-c730-4c60-97ce-f76495b990a8\" (UID: \"ee26e2d8-c730-4c60-97ce-f76495b990a8\") "
Oct 30 20:15:13.467423 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.467375    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee26e2d8-c730-4c60-97ce-f76495b990a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee26e2d8-c730-4c60-97ce-f76495b990a8-maas-api-service-ca\") pod \"ee26e2d8-c730-4c60-97ce-f76495b990a8\" (UID: \"ee26e2d8-c730-4c60-97ce-f76495b990a8\") "
Oct 30 20:15:13.467762 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.467724    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee26e2d8-c730-4c60-97ce-f76495b990a8-maas-api-service-ca" pod "ee26e2d8-c730-4c60-97ce-f76495b990a8" (UID: "ee26e2d8-c730-4c60-97ce-f76495b990a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:13.467947 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.467854    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee26e2d8-c730-4c60-97ce-f76495b990a8-maas-api-service-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:13.469603 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.469578    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee26e2d8-c730-4c60-97ce-f76495b990a8-kube-api-access-9vx7t" pod "ee26e2d8-c730-4c60-97ce-f76495b990a8" (UID: "ee26e2d8-c730-4c60-97ce-f76495b990a8"). InnerVolumeSpecName "kube-api-access-9vx7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:13.517624 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.517599    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x"
Oct 30 20:15:13.517773 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.517589    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x" event={"ID":"ee26e2d8-c730-4c60-97ce-f76495b990a8","Type":"ContainerDied","Data":"0db182fb471b053f6edfe99dd49ab95b0aad219f43d5381f4d201210c9af6f39"}
Oct 30 20:15:13.517773 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.517734    4823 scope.go:122] "RemoveContainer" containerID="8f99d7ce1f6439e62fdbd587f7a64097155fb28eaada5e12d5ed03b9755fc79c"
Oct 30 20:15:13.549101 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.549065    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x"]
Oct 30 20:15:13.552946 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.552911    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h" podUID="43b70bdc-bb09-4baa-ac3c-dbbefb3adf27" containerName="main" containerID="cri-o://daf4d078c05165ed44d91977998dba86f413ded0fd8942be8e3a12e0755f3ce3" gracePeriod=45
Oct 30 20:15:13.565674 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.565613    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-274922-29889855-xbx9x"]
Oct 30 20:15:13.572111 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.572074    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vx7t\" (UniqueName: \"kubernetes.io/projected/ee26e2d8-c730-4c60-97ce-f76495b990a8-kube-api-access-9vx7t\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:13.815878 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.815843    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:15:13.976634 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.976598    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-tls-certs\" (UniqueName: \"kubernetes.io/secret/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-tls-certs\") pod \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") "
Oct 30 20:15:13.976822 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.976693    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-kube-api-access-vwkxn\" (UniqueName: \"kubernetes.io/projected/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-kube-api-access-vwkxn\") pod \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") "
Oct 30 20:15:13.976822 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.976787    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-model-cache\") pod \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") "
Oct 30 20:15:13.977023 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.977004    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-home\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-home\") pod \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") "
Oct 30 20:15:13.977081 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.977062    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-dshm\") pod \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") "
Oct 30 20:15:13.977133 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.977078    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-model-cache" pod "43b70bdc-bb09-4baa-ac3c-dbbefb3adf27" (UID: "43b70bdc-bb09-4baa-ac3c-dbbefb3adf27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:13.977133 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.977105    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-tmp-dir\") pod \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\" (UID: \"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27\") "
Oct 30 20:15:13.977322 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.977271    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-home" pod "43b70bdc-bb09-4baa-ac3c-dbbefb3adf27" (UID: "43b70bdc-bb09-4baa-ac3c-dbbefb3adf27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:13.977416 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.977399    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-tmp-dir" pod "43b70bdc-bb09-4baa-ac3c-dbbefb3adf27" (UID: "43b70bdc-bb09-4baa-ac3c-dbbefb3adf27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:13.977723 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.977703    4823 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-home\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:13.977764 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.977731    4823 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-tmp-dir\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:13.977764 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.977746    4823 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-model-cache\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:13.979170 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.979146    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-kube-api-access-vwkxn" pod "43b70bdc-bb09-4baa-ac3c-dbbefb3adf27" (UID: "43b70bdc-bb09-4baa-ac3c-dbbefb3adf27"). InnerVolumeSpecName "kube-api-access-vwkxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:13.979170 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.979158    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-tls-certs" pod "43b70bdc-bb09-4baa-ac3c-dbbefb3adf27" (UID: "43b70bdc-bb09-4baa-ac3c-dbbefb3adf27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:13.979282 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:13.979152    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-dshm" pod "43b70bdc-bb09-4baa-ac3c-dbbefb3adf27" (UID: "43b70bdc-bb09-4baa-ac3c-dbbefb3adf27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:14.079028 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:14.078954    4823 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-dshm\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:14.079028 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:14.078980    4823 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-tls-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:14.079028 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:14.078991    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwkxn\" (UniqueName: \"kubernetes.io/projected/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27-kube-api-access-vwkxn\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:14.278956 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:14.278920    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee26e2d8-c730-4c60-97ce-f76495b990a8" path="/var/lib/kubelet/pods/ee26e2d8-c730-4c60-97ce-f76495b990a8/volumes"
Oct 30 20:15:14.526015 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:14.525973    4823 generic.go:356] "Generic (PLEG): container finished" podID="43b70bdc-bb09-4baa-ac3c-dbbefb3adf27" containerID="daf4d078c05165ed44d91977998dba86f413ded0fd8942be8e3a12e0755f3ce3" exitCode=0
Oct 30 20:15:14.526246 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:14.526023    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"
Oct 30 20:15:14.526246 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:14.526065    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h" event={"ID":"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27","Type":"ContainerDied","Data":"daf4d078c05165ed44d91977998dba86f413ded0fd8942be8e3a12e0755f3ce3"}
Oct 30 20:15:14.526246 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:14.526121    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h" event={"ID":"43b70bdc-bb09-4baa-ac3c-dbbefb3adf27","Type":"ContainerDied","Data":"6aa24e9c8ce68d1c0301f9ac2b54b1a20c8a428ca7200b7f5bbc7f1db39ba8f7"}
Oct 30 20:15:14.526246 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:14.526150    4823 scope.go:122] "RemoveContainer" containerID="daf4d078c05165ed44d91977998dba86f413ded0fd8942be8e3a12e0755f3ce3"
Oct 30 20:15:14.536771 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:14.536748    4823 scope.go:122] "RemoveContainer" containerID="daf4d078c05165ed44d91977998dba86f413ded0fd8942be8e3a12e0755f3ce3"
Oct 30 20:15:14.537091 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:15:14.537064    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf4d078c05165ed44d91977998dba86f413ded0fd8942be8e3a12e0755f3ce3\": container with ID starting with daf4d078c05165ed44d91977998dba86f413ded0fd8942be8e3a12e0755f3ce3 not found: ID does not exist" containerID="daf4d078c05165ed44d91977998dba86f413ded0fd8942be8e3a12e0755f3ce3"
Oct 30 20:15:14.537200 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:14.537098    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf4d078c05165ed44d91977998dba86f413ded0fd8942be8e3a12e0755f3ce3"} err="failed to get container status \"daf4d078c05165ed44d91977998dba86f413ded0fd8942be8e3a12e0755f3ce3\": rpc error: code = NotFound desc = could not find container \"daf4d078c05165ed44d91977998dba86f413ded0fd8942be8e3a12e0755f3ce3\": container with ID starting with daf4d078c05165ed44d91977998dba86f413ded0fd8942be8e3a12e0755f3ce3 not found: ID does not exist"
Oct 30 20:15:14.545342 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:14.545319    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"]
Oct 30 20:15:14.550964 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:14.550935    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-274922/auth-test-model-274922-kserve-5c748fcb4f-9bz9h"]
Oct 30 20:15:15.271686 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:15.271651    4823 scope.go:122] "RemoveContainer" containerID="d16f776a23f997a661b719037ad0053b2403e447f7a6371ee471a342ac59bdcb"
Oct 30 20:15:15.910713 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:15.910679    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:15:16.280015 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:16.279986    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b70bdc-bb09-4baa-ac3c-dbbefb3adf27" path="/var/lib/kubelet/pods/43b70bdc-bb09-4baa-ac3c-dbbefb3adf27/volumes"
Oct 30 20:15:16.545134 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:16.545053    4823 generic.go:356] "Generic (PLEG): container finished" podID="b3a911c9-4eda-46bd-a390-76b95d5e2c60" containerID="2c83538b1304d91781aa9d7ae0452f6987635d0aa25a2f51288eb1603c2ad8f3" exitCode=60
Oct 30 20:15:16.545319 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:16.545129    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph" event={"ID":"b3a911c9-4eda-46bd-a390-76b95d5e2c60","Type":"ContainerDied","Data":"2c83538b1304d91781aa9d7ae0452f6987635d0aa25a2f51288eb1603c2ad8f3"}
Oct 30 20:15:16.545319 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:16.545191    4823 scope.go:122] "RemoveContainer" containerID="d16f776a23f997a661b719037ad0053b2403e447f7a6371ee471a342ac59bdcb"
Oct 30 20:15:16.545583 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:16.545566    4823 scope.go:122] "RemoveContainer" containerID="2c83538b1304d91781aa9d7ae0452f6987635d0aa25a2f51288eb1603c2ad8f3"
Oct 30 20:15:16.545929 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:15:16.545910    4823 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph_odh-ai-gateway-infra(b3a911c9-4eda-46bd-a390-76b95d5e2c60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph" podUID="b3a911c9-4eda-46bd-a390-76b95d5e2c60"
Oct 30 20:15:17.576812 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:17.576778    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph"]
Oct 30 20:15:17.725537 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:17.725494    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"]
Oct 30 20:15:17.726023 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:17.725700    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd" podUID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" containerName="istio-proxy" containerID="cri-o://32c425d21eb4fa1e252b330c4f0cc54e85f51727ae1fb31d13cbf5595da88b46" gracePeriod=30
Oct 30 20:15:17.760412 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:17.760379    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph"
Oct 30 20:15:17.923274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:17.923240    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3a911c9-4eda-46bd-a390-76b95d5e2c60-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a911c9-4eda-46bd-a390-76b95d5e2c60-maas-api-service-ca\") pod \"b3a911c9-4eda-46bd-a390-76b95d5e2c60\" (UID: \"b3a911c9-4eda-46bd-a390-76b95d5e2c60\") "
Oct 30 20:15:17.923482 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:17.923330    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a911c9-4eda-46bd-a390-76b95d5e2c60-kube-api-access-vn9lz\" (UniqueName: \"kubernetes.io/projected/b3a911c9-4eda-46bd-a390-76b95d5e2c60-kube-api-access-vn9lz\") pod \"b3a911c9-4eda-46bd-a390-76b95d5e2c60\" (UID: \"b3a911c9-4eda-46bd-a390-76b95d5e2c60\") "
Oct 30 20:15:17.923638 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:17.923612    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a911c9-4eda-46bd-a390-76b95d5e2c60-maas-api-service-ca" pod "b3a911c9-4eda-46bd-a390-76b95d5e2c60" (UID: "b3a911c9-4eda-46bd-a390-76b95d5e2c60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:17.925449 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:17.925426    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a911c9-4eda-46bd-a390-76b95d5e2c60-kube-api-access-vn9lz" pod "b3a911c9-4eda-46bd-a390-76b95d5e2c60" (UID: "b3a911c9-4eda-46bd-a390-76b95d5e2c60"). InnerVolumeSpecName "kube-api-access-vn9lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:18.024432 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:18.024399    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a911c9-4eda-46bd-a390-76b95d5e2c60-maas-api-service-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.024432 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:18.024429    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn9lz\" (UniqueName: \"kubernetes.io/projected/b3a911c9-4eda-46bd-a390-76b95d5e2c60-kube-api-access-vn9lz\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.100737 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:18.100701    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:15:18.100939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:18.100754    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd" podUID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:15:18.564408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:18.564328    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph"
Oct 30 20:15:18.564408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:18.564357    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph" event={"ID":"b3a911c9-4eda-46bd-a390-76b95d5e2c60","Type":"ContainerDied","Data":"e85457ac42853099ae441569bfe926ce39e2fd1584cf729356c9cf32e48fe21d"}
Oct 30 20:15:18.564643 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:18.564427    4823 scope.go:122] "RemoveContainer" containerID="2c83538b1304d91781aa9d7ae0452f6987635d0aa25a2f51288eb1603c2ad8f3"
Oct 30 20:15:18.581849 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:18.581824    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph"]
Oct 30 20:15:18.587513 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:18.587491    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889855-5kvph"]
Oct 30 20:15:20.306781 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:20.306744    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3a911c9-4eda-46bd-a390-76b95d5e2c60" path="/var/lib/kubelet/pods/b3a911c9-4eda-46bd-a390-76b95d5e2c60/volumes"
Oct 30 20:15:22.316414 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:22.316374    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"]
Oct 30 20:15:22.316939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:22.316666    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj" podUID="326b0731-4c61-494d-9de4-0ad913784cf1" containerName="istio-proxy" containerID="cri-o://f93d2ed16d502d9315fbd5ffde4cfb6804866de5813ba0b705373fed1082c994" gracePeriod=30
Oct 30 20:15:23.004483 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.004268    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 20:15:23.074273 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.074239    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-workload-socket\") pod \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") "
Oct 30 20:15:23.074471 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.074303    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-podinfo\") pod \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") "
Oct 30 20:15:23.074471 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.074346    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-envoy\") pod \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") "
Oct 30 20:15:23.074471 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.074380    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-token\" (UniqueName: \"kubernetes.io/projected/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-token\") pod \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") "
Oct 30 20:15:23.074471 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.074409    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-data\") pod \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") "
Oct 30 20:15:23.074471 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.074446    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istiod-ca-cert\") pod \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") "
Oct 30 20:15:23.074755 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.074479    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-credential-socket\") pod \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") "
Oct 30 20:15:23.074755 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.074544    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-workload-socket" pod "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" (UID: "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:23.074755 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.074562    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-kube-api-access-d2cps\" (UniqueName: \"kubernetes.io/projected/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-kube-api-access-d2cps\") pod \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") "
Oct 30 20:15:23.074755 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.074612    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-workload-certs\") pod \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\" (UID: \"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0\") "
Oct 30 20:15:23.075159 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.075109    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.075355 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.075313    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-workload-certs" pod "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" (UID: "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:23.075441 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.075379    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-data" pod "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" (UID: "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:23.075592 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.075568    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istiod-ca-cert" pod "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" (UID: "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:23.075672 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.075611    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-credential-socket" pod "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" (UID: "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:23.077384 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.077361    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-token" pod "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" (UID: "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:23.077479 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.077394    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-kube-api-access-d2cps" pod "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" (UID: "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0"). InnerVolumeSpecName "kube-api-access-d2cps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:23.077479 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.077423    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-envoy" pod "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" (UID: "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:23.078922 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.078894    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-podinfo" pod "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" (UID: "bda5b0d8-6d21-4ff4-a860-d0f63f912ee0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:23.176087 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.176052    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2cps\" (UniqueName: \"kubernetes.io/projected/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-kube-api-access-d2cps\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.176087 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.176082    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.176087 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.176092    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.176374 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.176100    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.176374 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.176109    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.176374 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.176117    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.176374 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.176125    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.176374 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.176133    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.606974 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.606942    4823 generic.go:356] "Generic (PLEG): container finished" podID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" containerID="32c425d21eb4fa1e252b330c4f0cc54e85f51727ae1fb31d13cbf5595da88b46" exitCode=0
Oct 30 20:15:23.607609 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.607003    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"
Oct 30 20:15:23.607609 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.607030    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd" event={"ID":"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0","Type":"ContainerDied","Data":"32c425d21eb4fa1e252b330c4f0cc54e85f51727ae1fb31d13cbf5595da88b46"}
Oct 30 20:15:23.607609 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.607099    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd" event={"ID":"bda5b0d8-6d21-4ff4-a860-d0f63f912ee0","Type":"ContainerDied","Data":"4f7d73fe6ab8962578a58720f75ac10ca6f31b503ce071c6b74b33d3367d42db"}
Oct 30 20:15:23.607609 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.607123    4823 scope.go:122] "RemoveContainer" containerID="32c425d21eb4fa1e252b330c4f0cc54e85f51727ae1fb31d13cbf5595da88b46"
Oct 30 20:15:23.618561 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.618541    4823 scope.go:122] "RemoveContainer" containerID="c9ff1fae34a53e9dd981298a9664a6cf4fa8b421897d3b40e80b50625cd56853"
Oct 30 20:15:23.628087 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.628065    4823 scope.go:122] "RemoveContainer" containerID="32c425d21eb4fa1e252b330c4f0cc54e85f51727ae1fb31d13cbf5595da88b46"
Oct 30 20:15:23.628720 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:15:23.628694    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c425d21eb4fa1e252b330c4f0cc54e85f51727ae1fb31d13cbf5595da88b46\": container with ID starting with 32c425d21eb4fa1e252b330c4f0cc54e85f51727ae1fb31d13cbf5595da88b46 not found: ID does not exist" containerID="32c425d21eb4fa1e252b330c4f0cc54e85f51727ae1fb31d13cbf5595da88b46"
Oct 30 20:15:23.628828 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.628724    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c425d21eb4fa1e252b330c4f0cc54e85f51727ae1fb31d13cbf5595da88b46"} err="failed to get container status \"32c425d21eb4fa1e252b330c4f0cc54e85f51727ae1fb31d13cbf5595da88b46\": rpc error: code = NotFound desc = could not find container \"32c425d21eb4fa1e252b330c4f0cc54e85f51727ae1fb31d13cbf5595da88b46\": container with ID starting with 32c425d21eb4fa1e252b330c4f0cc54e85f51727ae1fb31d13cbf5595da88b46 not found: ID does not exist"
Oct 30 20:15:23.628828 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.628741    4823 scope.go:122] "RemoveContainer" containerID="c9ff1fae34a53e9dd981298a9664a6cf4fa8b421897d3b40e80b50625cd56853"
Oct 30 20:15:23.629123 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:15:23.629103    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ff1fae34a53e9dd981298a9664a6cf4fa8b421897d3b40e80b50625cd56853\": container with ID starting with c9ff1fae34a53e9dd981298a9664a6cf4fa8b421897d3b40e80b50625cd56853 not found: ID does not exist" containerID="c9ff1fae34a53e9dd981298a9664a6cf4fa8b421897d3b40e80b50625cd56853"
Oct 30 20:15:23.629214 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.629129    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ff1fae34a53e9dd981298a9664a6cf4fa8b421897d3b40e80b50625cd56853"} err="failed to get container status \"c9ff1fae34a53e9dd981298a9664a6cf4fa8b421897d3b40e80b50625cd56853\": rpc error: code = NotFound desc = could not find container \"c9ff1fae34a53e9dd981298a9664a6cf4fa8b421897d3b40e80b50625cd56853\": container with ID starting with c9ff1fae34a53e9dd981298a9664a6cf4fa8b421897d3b40e80b50625cd56853 not found: ID does not exist"
Oct 30 20:15:23.632011 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.631988    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"]
Oct 30 20:15:23.638259 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:23.638238    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-627xd"]
Oct 30 20:15:24.280824 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:24.280786    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" path="/var/lib/kubelet/pods/bda5b0d8-6d21-4ff4-a860-d0f63f912ee0/volumes"
Oct 30 20:15:27.575730 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.575699    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 20:15:27.635070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.635035    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-istio-data\") pod \"326b0731-4c61-494d-9de4-0ad913784cf1\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") "
Oct 30 20:15:27.635263 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.635085    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-credential-socket\") pod \"326b0731-4c61-494d-9de4-0ad913784cf1\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") "
Oct 30 20:15:27.635263 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.635163    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326b0731-4c61-494d-9de4-0ad913784cf1-kube-api-access-q5bxb\" (UniqueName: \"kubernetes.io/projected/326b0731-4c61-494d-9de4-0ad913784cf1-kube-api-access-q5bxb\") pod \"326b0731-4c61-494d-9de4-0ad913784cf1\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") "
Oct 30 20:15:27.635263 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.635230    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-workload-certs\") pod \"326b0731-4c61-494d-9de4-0ad913784cf1\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") "
Oct 30 20:15:27.635477 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.635308    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-workload-socket\") pod \"326b0731-4c61-494d-9de4-0ad913784cf1\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") "
Oct 30 20:15:27.635477 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.635347    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/326b0731-4c61-494d-9de4-0ad913784cf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326b0731-4c61-494d-9de4-0ad913784cf1-istio-podinfo\") pod \"326b0731-4c61-494d-9de4-0ad913784cf1\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") "
Oct 30 20:15:27.635477 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.635370    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-istio-data" pod "326b0731-4c61-494d-9de4-0ad913784cf1" (UID: "326b0731-4c61-494d-9de4-0ad913784cf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.635477 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.635417    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/326b0731-4c61-494d-9de4-0ad913784cf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326b0731-4c61-494d-9de4-0ad913784cf1-istiod-ca-cert\") pod \"326b0731-4c61-494d-9de4-0ad913784cf1\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") "
Oct 30 20:15:27.635477 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.635471    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326b0731-4c61-494d-9de4-0ad913784cf1-istio-token\" (UniqueName: \"kubernetes.io/projected/326b0731-4c61-494d-9de4-0ad913784cf1-istio-token\") pod \"326b0731-4c61-494d-9de4-0ad913784cf1\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") "
Oct 30 20:15:27.635759 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.635527    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-istio-envoy\") pod \"326b0731-4c61-494d-9de4-0ad913784cf1\" (UID: \"326b0731-4c61-494d-9de4-0ad913784cf1\") "
Oct 30 20:15:27.635759 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.635702    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-workload-socket" pod "326b0731-4c61-494d-9de4-0ad913784cf1" (UID: "326b0731-4c61-494d-9de4-0ad913784cf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.635759 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.635721    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-workload-certs" pod "326b0731-4c61-494d-9de4-0ad913784cf1" (UID: "326b0731-4c61-494d-9de4-0ad913784cf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.636129 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.636100    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-credential-socket" pod "326b0731-4c61-494d-9de4-0ad913784cf1" (UID: "326b0731-4c61-494d-9de4-0ad913784cf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.636274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.636158    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326b0731-4c61-494d-9de4-0ad913784cf1-istiod-ca-cert" pod "326b0731-4c61-494d-9de4-0ad913784cf1" (UID: "326b0731-4c61-494d-9de4-0ad913784cf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:27.636415 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.636391    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.636502 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.636422    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.636502 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.636438    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326b0731-4c61-494d-9de4-0ad913784cf1-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.636502 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.636451    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.636502 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.636465    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.638201 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.638175    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-istio-envoy" pod "326b0731-4c61-494d-9de4-0ad913784cf1" (UID: "326b0731-4c61-494d-9de4-0ad913784cf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.638346 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.638302    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/326b0731-4c61-494d-9de4-0ad913784cf1-istio-podinfo" pod "326b0731-4c61-494d-9de4-0ad913784cf1" (UID: "326b0731-4c61-494d-9de4-0ad913784cf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:27.639002 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.638973    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326b0731-4c61-494d-9de4-0ad913784cf1-kube-api-access-q5bxb" pod "326b0731-4c61-494d-9de4-0ad913784cf1" (UID: "326b0731-4c61-494d-9de4-0ad913784cf1"). InnerVolumeSpecName "kube-api-access-q5bxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:27.639111 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.639014    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326b0731-4c61-494d-9de4-0ad913784cf1-istio-token" pod "326b0731-4c61-494d-9de4-0ad913784cf1" (UID: "326b0731-4c61-494d-9de4-0ad913784cf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:27.639983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.639957    4823 generic.go:356] "Generic (PLEG): container finished" podID="326b0731-4c61-494d-9de4-0ad913784cf1" containerID="f93d2ed16d502d9315fbd5ffde4cfb6804866de5813ba0b705373fed1082c994" exitCode=0
Oct 30 20:15:27.640111 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.640027    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"
Oct 30 20:15:27.640111 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.640035    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj" event={"ID":"326b0731-4c61-494d-9de4-0ad913784cf1","Type":"ContainerDied","Data":"f93d2ed16d502d9315fbd5ffde4cfb6804866de5813ba0b705373fed1082c994"}
Oct 30 20:15:27.640245 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.640144    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj" event={"ID":"326b0731-4c61-494d-9de4-0ad913784cf1","Type":"ContainerDied","Data":"9acadb7bb8cd99c137ce05e6c2e4d560ab2321bc67686888c9f48c81ba5e7f54"}
Oct 30 20:15:27.640245 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.640176    4823 scope.go:122] "RemoveContainer" containerID="f93d2ed16d502d9315fbd5ffde4cfb6804866de5813ba0b705373fed1082c994"
Oct 30 20:15:27.668854 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.668830    4823 scope.go:122] "RemoveContainer" containerID="66f4a5087cb706b96c414e45c1189fb615fdd35058c63480c8d1b9a669e3350c"
Oct 30 20:15:27.678460 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.678438    4823 scope.go:122] "RemoveContainer" containerID="f93d2ed16d502d9315fbd5ffde4cfb6804866de5813ba0b705373fed1082c994"
Oct 30 20:15:27.678798 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:15:27.678766    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93d2ed16d502d9315fbd5ffde4cfb6804866de5813ba0b705373fed1082c994\": container with ID starting with f93d2ed16d502d9315fbd5ffde4cfb6804866de5813ba0b705373fed1082c994 not found: ID does not exist" containerID="f93d2ed16d502d9315fbd5ffde4cfb6804866de5813ba0b705373fed1082c994"
Oct 30 20:15:27.678939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.678802    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93d2ed16d502d9315fbd5ffde4cfb6804866de5813ba0b705373fed1082c994"} err="failed to get container status \"f93d2ed16d502d9315fbd5ffde4cfb6804866de5813ba0b705373fed1082c994\": rpc error: code = NotFound desc = could not find container \"f93d2ed16d502d9315fbd5ffde4cfb6804866de5813ba0b705373fed1082c994\": container with ID starting with f93d2ed16d502d9315fbd5ffde4cfb6804866de5813ba0b705373fed1082c994 not found: ID does not exist"
Oct 30 20:15:27.678939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.678820    4823 scope.go:122] "RemoveContainer" containerID="66f4a5087cb706b96c414e45c1189fb615fdd35058c63480c8d1b9a669e3350c"
Oct 30 20:15:27.679272 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:15:27.679247    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f4a5087cb706b96c414e45c1189fb615fdd35058c63480c8d1b9a669e3350c\": container with ID starting with 66f4a5087cb706b96c414e45c1189fb615fdd35058c63480c8d1b9a669e3350c not found: ID does not exist" containerID="66f4a5087cb706b96c414e45c1189fb615fdd35058c63480c8d1b9a669e3350c"
Oct 30 20:15:27.679344 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.679279    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f4a5087cb706b96c414e45c1189fb615fdd35058c63480c8d1b9a669e3350c"} err="failed to get container status \"66f4a5087cb706b96c414e45c1189fb615fdd35058c63480c8d1b9a669e3350c\": rpc error: code = NotFound desc = could not find container \"66f4a5087cb706b96c414e45c1189fb615fdd35058c63480c8d1b9a669e3350c\": container with ID starting with 66f4a5087cb706b96c414e45c1189fb615fdd35058c63480c8d1b9a669e3350c not found: ID does not exist"
Oct 30 20:15:27.682554 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.682531    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"]
Oct 30 20:15:27.686572 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.686552    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-274922-openshift-default-7d7d5f5869-tmtcj"]
Oct 30 20:15:27.737662 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.737637    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326b0731-4c61-494d-9de4-0ad913784cf1-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.737662 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.737661    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326b0731-4c61-494d-9de4-0ad913784cf1-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.737662 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.737673    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5bxb\" (UniqueName: \"kubernetes.io/projected/326b0731-4c61-494d-9de4-0ad913784cf1-kube-api-access-q5bxb\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.737917 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:27.737683    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326b0731-4c61-494d-9de4-0ad913784cf1-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:28.279826 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:28.279782    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="326b0731-4c61-494d-9de4-0ad913784cf1" path="/var/lib/kubelet/pods/326b0731-4c61-494d-9de4-0ad913784cf1/volumes"
Oct 30 20:15:32.015489 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:15:32.015448    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:15:36.163131 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:36.163097    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:15:42.521425 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:42.521345    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"]
Oct 30 20:15:42.521982 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:42.521579    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth" podUID="d69555e5-864f-459f-bb32-a8c4fd803b51" containerName="istio-proxy" containerID="cri-o://e565066f28ab9679fe38b0742af87fca841df578e1624afcc398652226c381de" gracePeriod=30
Oct 30 20:15:42.563525 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:42.563481    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"]
Oct 30 20:15:42.566247 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:42.563902    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25" podUID="778e153e-48af-4716-9fb2-2095e02c7bc2" containerName="istio-proxy" containerID="cri-o://78552bdf49410b8849d4f370fe26108fc01e2dbcecd7ca43f12084f2f14c4628" gracePeriod=30
Oct 30 20:15:42.568891 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:42.568849    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"]
Oct 30 20:15:42.569300 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:42.569251    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv" podUID="251fadc1-f1f5-431d-9286-5bd16b99733e" containerName="istio-proxy" containerID="cri-o://b8476dea9d1a0ee063810078624f0450a4f0b4a58a5db46c8f7cb63bcc3bb8e8" gracePeriod=30
Oct 30 20:15:47.664580 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.664547    4823 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:15:47.665029 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.664599    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25" podUID="778e153e-48af-4716-9fb2-2095e02c7bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Oct 30 20:15:47.823682 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.823648    4823 generic.go:356] "Generic (PLEG): container finished" podID="778e153e-48af-4716-9fb2-2095e02c7bc2" containerID="78552bdf49410b8849d4f370fe26108fc01e2dbcecd7ca43f12084f2f14c4628" exitCode=0
Oct 30 20:15:47.823803 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.823719    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25" event={"ID":"778e153e-48af-4716-9fb2-2095e02c7bc2","Type":"ContainerDied","Data":"78552bdf49410b8849d4f370fe26108fc01e2dbcecd7ca43f12084f2f14c4628"}
Oct 30 20:15:47.823803 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.823797    4823 scope.go:122] "RemoveContainer" containerID="a13408882b277f94c3e5ec6cc9f708b7f0d754940aa41676980403d80cfa3938"
Oct 30 20:15:47.826801 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.826774    4823 generic.go:356] "Generic (PLEG): container finished" podID="251fadc1-f1f5-431d-9286-5bd16b99733e" containerID="b8476dea9d1a0ee063810078624f0450a4f0b4a58a5db46c8f7cb63bcc3bb8e8" exitCode=0
Oct 30 20:15:47.826963 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.826942    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv" event={"ID":"251fadc1-f1f5-431d-9286-5bd16b99733e","Type":"ContainerDied","Data":"b8476dea9d1a0ee063810078624f0450a4f0b4a58a5db46c8f7cb63bcc3bb8e8"}
Oct 30 20:15:47.829836 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.829660    4823 generic.go:356] "Generic (PLEG): container finished" podID="d69555e5-864f-459f-bb32-a8c4fd803b51" containerID="e565066f28ab9679fe38b0742af87fca841df578e1624afcc398652226c381de" exitCode=0
Oct 30 20:15:47.829836 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.829754    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth" event={"ID":"d69555e5-864f-459f-bb32-a8c4fd803b51","Type":"ContainerDied","Data":"e565066f28ab9679fe38b0742af87fca841df578e1624afcc398652226c381de"}
Oct 30 20:15:47.857290 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.857262    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:15:47.929952 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.929927    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 20:15:47.932069 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.932049    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:15:47.973804 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.973767    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-workload-socket\") pod \"d69555e5-864f-459f-bb32-a8c4fd803b51\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") "
Oct 30 20:15:47.973804 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.973807    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-envoy\") pod \"d69555e5-864f-459f-bb32-a8c4fd803b51\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") "
Oct 30 20:15:47.974085 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.973848    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-token\" (UniqueName: \"kubernetes.io/projected/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-token\") pod \"d69555e5-864f-459f-bb32-a8c4fd803b51\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") "
Oct 30 20:15:47.974085 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.973928    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-podinfo\") pod \"d69555e5-864f-459f-bb32-a8c4fd803b51\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") "
Oct 30 20:15:47.974085 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.973990    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-data\") pod \"d69555e5-864f-459f-bb32-a8c4fd803b51\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") "
Oct 30 20:15:47.974294 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.974258    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-workload-socket" pod "d69555e5-864f-459f-bb32-a8c4fd803b51" (UID: "d69555e5-864f-459f-bb32-a8c4fd803b51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.974532 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.974506    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d69555e5-864f-459f-bb32-a8c4fd803b51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d69555e5-864f-459f-bb32-a8c4fd803b51-istiod-ca-cert\") pod \"d69555e5-864f-459f-bb32-a8c4fd803b51\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") "
Oct 30 20:15:47.974628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.974568    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-credential-socket\") pod \"d69555e5-864f-459f-bb32-a8c4fd803b51\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") "
Oct 30 20:15:47.974628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.974610    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69555e5-864f-459f-bb32-a8c4fd803b51-kube-api-access-vkcbn\" (UniqueName: \"kubernetes.io/projected/d69555e5-864f-459f-bb32-a8c4fd803b51-kube-api-access-vkcbn\") pod \"d69555e5-864f-459f-bb32-a8c4fd803b51\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") "
Oct 30 20:15:47.974740 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.974651    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-workload-certs\") pod \"d69555e5-864f-459f-bb32-a8c4fd803b51\" (UID: \"d69555e5-864f-459f-bb32-a8c4fd803b51\") "
Oct 30 20:15:47.974824 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.974805    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69555e5-864f-459f-bb32-a8c4fd803b51-istiod-ca-cert" pod "d69555e5-864f-459f-bb32-a8c4fd803b51" (UID: "d69555e5-864f-459f-bb32-a8c4fd803b51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:47.974924 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.974824    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-credential-socket" pod "d69555e5-864f-459f-bb32-a8c4fd803b51" (UID: "d69555e5-864f-459f-bb32-a8c4fd803b51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.975082 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.975062    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-workload-certs" pod "d69555e5-864f-459f-bb32-a8c4fd803b51" (UID: "d69555e5-864f-459f-bb32-a8c4fd803b51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.975156 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.974607    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-data" pod "d69555e5-864f-459f-bb32-a8c4fd803b51" (UID: "d69555e5-864f-459f-bb32-a8c4fd803b51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.975241 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.975223    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d69555e5-864f-459f-bb32-a8c4fd803b51-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:47.975302 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.975245    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:47.975302 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.975258    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:47.975302 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.975271    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:47.975302 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.975283    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:47.976549 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.976516    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-token" pod "d69555e5-864f-459f-bb32-a8c4fd803b51" (UID: "d69555e5-864f-459f-bb32-a8c4fd803b51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:47.977325 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.977299    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-envoy" pod "d69555e5-864f-459f-bb32-a8c4fd803b51" (UID: "d69555e5-864f-459f-bb32-a8c4fd803b51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:47.977403 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.977341    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69555e5-864f-459f-bb32-a8c4fd803b51-kube-api-access-vkcbn" pod "d69555e5-864f-459f-bb32-a8c4fd803b51" (UID: "d69555e5-864f-459f-bb32-a8c4fd803b51"). InnerVolumeSpecName "kube-api-access-vkcbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:47.977805 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:47.977789    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-podinfo" pod "d69555e5-864f-459f-bb32-a8c4fd803b51" (UID: "d69555e5-864f-459f-bb32-a8c4fd803b51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:48.032322 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.032279    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"]
Oct 30 20:15:48.041981 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.041951    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"]
Oct 30 20:15:48.075819 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.075736    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-workload-socket\") pod \"251fadc1-f1f5-431d-9286-5bd16b99733e\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") "
Oct 30 20:15:48.075819 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.075793    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-token\" (UniqueName: \"kubernetes.io/projected/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-token\") pod \"251fadc1-f1f5-431d-9286-5bd16b99733e\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") "
Oct 30 20:15:48.076084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.075818    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-workload-socket\") pod \"778e153e-48af-4716-9fb2-2095e02c7bc2\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") "
Oct 30 20:15:48.076084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.075887    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251fadc1-f1f5-431d-9286-5bd16b99733e-kube-api-access-8gsbs\" (UniqueName: \"kubernetes.io/projected/251fadc1-f1f5-431d-9286-5bd16b99733e-kube-api-access-8gsbs\") pod \"251fadc1-f1f5-431d-9286-5bd16b99733e\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") "
Oct 30 20:15:48.076084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.075917    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-envoy\") pod \"251fadc1-f1f5-431d-9286-5bd16b99733e\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") "
Oct 30 20:15:48.076084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.075952    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-podinfo\") pod \"778e153e-48af-4716-9fb2-2095e02c7bc2\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") "
Oct 30 20:15:48.076084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.075981    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-envoy\") pod \"778e153e-48af-4716-9fb2-2095e02c7bc2\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") "
Oct 30 20:15:48.076084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076016    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-data\") pod \"251fadc1-f1f5-431d-9286-5bd16b99733e\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") "
Oct 30 20:15:48.076084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076066    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-workload-certs\") pod \"251fadc1-f1f5-431d-9286-5bd16b99733e\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") "
Oct 30 20:15:48.076463 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076121    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-workload-socket" pod "778e153e-48af-4716-9fb2-2095e02c7bc2" (UID: "778e153e-48af-4716-9fb2-2095e02c7bc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:48.076463 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076179    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-credential-socket\") pod \"251fadc1-f1f5-431d-9286-5bd16b99733e\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") "
Oct 30 20:15:48.076463 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076243    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-podinfo\") pod \"251fadc1-f1f5-431d-9286-5bd16b99733e\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") "
Oct 30 20:15:48.076463 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076271    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-data\") pod \"778e153e-48af-4716-9fb2-2095e02c7bc2\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") "
Oct 30 20:15:48.076463 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076303    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251fadc1-f1f5-431d-9286-5bd16b99733e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251fadc1-f1f5-431d-9286-5bd16b99733e-istiod-ca-cert\") pod \"251fadc1-f1f5-431d-9286-5bd16b99733e\" (UID: \"251fadc1-f1f5-431d-9286-5bd16b99733e\") "
Oct 30 20:15:48.076463 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076339    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-workload-socket" pod "251fadc1-f1f5-431d-9286-5bd16b99733e" (UID: "251fadc1-f1f5-431d-9286-5bd16b99733e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:48.076463 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076362    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-data" pod "251fadc1-f1f5-431d-9286-5bd16b99733e" (UID: "251fadc1-f1f5-431d-9286-5bd16b99733e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:48.076463 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076350    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-credential-socket\") pod \"778e153e-48af-4716-9fb2-2095e02c7bc2\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") "
Oct 30 20:15:48.076463 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076444    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/778e153e-48af-4716-9fb2-2095e02c7bc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778e153e-48af-4716-9fb2-2095e02c7bc2-istiod-ca-cert\") pod \"778e153e-48af-4716-9fb2-2095e02c7bc2\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") "
Oct 30 20:15:48.076975 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076528    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-workload-certs\") pod \"778e153e-48af-4716-9fb2-2095e02c7bc2\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") "
Oct 30 20:15:48.076975 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076549    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-token\" (UniqueName: \"kubernetes.io/projected/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-token\") pod \"778e153e-48af-4716-9fb2-2095e02c7bc2\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") "
Oct 30 20:15:48.076975 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076566    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-credential-socket" pod "778e153e-48af-4716-9fb2-2095e02c7bc2" (UID: "778e153e-48af-4716-9fb2-2095e02c7bc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:48.076975 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076603    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778e153e-48af-4716-9fb2-2095e02c7bc2-kube-api-access-cdk56\" (UniqueName: \"kubernetes.io/projected/778e153e-48af-4716-9fb2-2095e02c7bc2-kube-api-access-cdk56\") pod \"778e153e-48af-4716-9fb2-2095e02c7bc2\" (UID: \"778e153e-48af-4716-9fb2-2095e02c7bc2\") "
Oct 30 20:15:48.076975 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076735    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-workload-certs" pod "251fadc1-f1f5-431d-9286-5bd16b99733e" (UID: "251fadc1-f1f5-431d-9286-5bd16b99733e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:48.076975 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076765    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-data" pod "778e153e-48af-4716-9fb2-2095e02c7bc2" (UID: "778e153e-48af-4716-9fb2-2095e02c7bc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:48.077306 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.076987    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-workload-certs" pod "778e153e-48af-4716-9fb2-2095e02c7bc2" (UID: "778e153e-48af-4716-9fb2-2095e02c7bc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:48.077306 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.077127    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251fadc1-f1f5-431d-9286-5bd16b99733e-istiod-ca-cert" pod "251fadc1-f1f5-431d-9286-5bd16b99733e" (UID: "251fadc1-f1f5-431d-9286-5bd16b99733e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:48.077404 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.077219    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.077404 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.077339    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.077404 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.077354    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.077404 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.077364    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.077404 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.077375    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.077404 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.077275    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-credential-socket" pod "251fadc1-f1f5-431d-9286-5bd16b99733e" (UID: "251fadc1-f1f5-431d-9286-5bd16b99733e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:48.077404 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.077391    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkcbn\" (UniqueName: \"kubernetes.io/projected/d69555e5-864f-459f-bb32-a8c4fd803b51-kube-api-access-vkcbn\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.077404 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.077293    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778e153e-48af-4716-9fb2-2095e02c7bc2-istiod-ca-cert" pod "778e153e-48af-4716-9fb2-2095e02c7bc2" (UID: "778e153e-48af-4716-9fb2-2095e02c7bc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:48.077404 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.077402    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.077732 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.077439    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.077732 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.077459    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.077732 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.077473    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.077732 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.077488    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d69555e5-864f-459f-bb32-a8c4fd803b51-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.079121 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.079094    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-envoy" pod "778e153e-48af-4716-9fb2-2095e02c7bc2" (UID: "778e153e-48af-4716-9fb2-2095e02c7bc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:48.079277 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.079129    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251fadc1-f1f5-431d-9286-5bd16b99733e-kube-api-access-8gsbs" pod "251fadc1-f1f5-431d-9286-5bd16b99733e" (UID: "251fadc1-f1f5-431d-9286-5bd16b99733e"). InnerVolumeSpecName "kube-api-access-8gsbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:48.079277 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.079196    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-token" pod "778e153e-48af-4716-9fb2-2095e02c7bc2" (UID: "778e153e-48af-4716-9fb2-2095e02c7bc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:48.079277 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.079257    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-token" pod "251fadc1-f1f5-431d-9286-5bd16b99733e" (UID: "251fadc1-f1f5-431d-9286-5bd16b99733e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:48.079473 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.079278    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-envoy" pod "251fadc1-f1f5-431d-9286-5bd16b99733e" (UID: "251fadc1-f1f5-431d-9286-5bd16b99733e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:48.079634 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.079607    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-podinfo" pod "251fadc1-f1f5-431d-9286-5bd16b99733e" (UID: "251fadc1-f1f5-431d-9286-5bd16b99733e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:48.080461 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.080433    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778e153e-48af-4716-9fb2-2095e02c7bc2-kube-api-access-cdk56" pod "778e153e-48af-4716-9fb2-2095e02c7bc2" (UID: "778e153e-48af-4716-9fb2-2095e02c7bc2"). InnerVolumeSpecName "kube-api-access-cdk56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:48.080749 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.080730    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-podinfo" pod "778e153e-48af-4716-9fb2-2095e02c7bc2" (UID: "778e153e-48af-4716-9fb2-2095e02c7bc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:48.179068 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.179035    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.179068 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.179067    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdk56\" (UniqueName: \"kubernetes.io/projected/778e153e-48af-4716-9fb2-2095e02c7bc2-kube-api-access-cdk56\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.179068 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.179078    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.179306 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.179087    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gsbs\" (UniqueName: \"kubernetes.io/projected/251fadc1-f1f5-431d-9286-5bd16b99733e-kube-api-access-8gsbs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.179306 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.179096    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.179306 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.179104    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.179306 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.179112    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778e153e-48af-4716-9fb2-2095e02c7bc2-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.179306 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.179121    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fadc1-f1f5-431d-9286-5bd16b99733e-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.179306 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.179129    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251fadc1-f1f5-431d-9286-5bd16b99733e-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.179306 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.179136    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251fadc1-f1f5-431d-9286-5bd16b99733e-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.179306 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.179144    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778e153e-48af-4716-9fb2-2095e02c7bc2-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:48.838251 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.838154    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25" event={"ID":"778e153e-48af-4716-9fb2-2095e02c7bc2","Type":"ContainerDied","Data":"1a88b234f4e71ade191b65268336b577c4d3920ec8948c01a4dcb4f4fc92a9f8"}
Oct 30 20:15:48.838251 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.838172    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"
Oct 30 20:15:48.838251 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.838214    4823 scope.go:122] "RemoveContainer" containerID="78552bdf49410b8849d4f370fe26108fc01e2dbcecd7ca43f12084f2f14c4628"
Oct 30 20:15:48.839787 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.839763    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"
Oct 30 20:15:48.839956 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.839797    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv" event={"ID":"251fadc1-f1f5-431d-9286-5bd16b99733e","Type":"ContainerDied","Data":"f49ac953c337a4b7a3fc3184a8ae40d46236a82accf065df862661f0969fe05e"}
Oct 30 20:15:48.841515 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.841493    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth" event={"ID":"d69555e5-864f-459f-bb32-a8c4fd803b51","Type":"ContainerDied","Data":"f087cbe92bc00050bb10a21ceb1f59da71f70e54c671ffb3854e858ba34f2518"}
Oct 30 20:15:48.841515 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.841529    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"
Oct 30 20:15:48.849962 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.849938    4823 scope.go:122] "RemoveContainer" containerID="b8476dea9d1a0ee063810078624f0450a4f0b4a58a5db46c8f7cb63bcc3bb8e8"
Oct 30 20:15:48.859337 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.859320    4823 scope.go:122] "RemoveContainer" containerID="e565066f28ab9679fe38b0742af87fca841df578e1624afcc398652226c381de"
Oct 30 20:15:48.864301 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.864280    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"]
Oct 30 20:15:48.870467 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.870445    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-6sb25"]
Oct 30 20:15:48.881580 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.881557    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"]
Oct 30 20:15:48.889281 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.889254    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-lpmfv"]
Oct 30 20:15:48.909913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.909877    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"]
Oct 30 20:15:48.914757 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:48.914728    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-2bpth"]
Oct 30 20:15:50.279208 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:50.279172    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251fadc1-f1f5-431d-9286-5bd16b99733e" path="/var/lib/kubelet/pods/251fadc1-f1f5-431d-9286-5bd16b99733e/volumes"
Oct 30 20:15:50.279835 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:50.279813    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="778e153e-48af-4716-9fb2-2095e02c7bc2" path="/var/lib/kubelet/pods/778e153e-48af-4716-9fb2-2095e02c7bc2/volumes"
Oct 30 20:15:50.280323 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:50.280302    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d69555e5-864f-459f-bb32-a8c4fd803b51" path="/var/lib/kubelet/pods/d69555e5-864f-459f-bb32-a8c4fd803b51/volumes"
Oct 30 20:15:53.313826 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:53.313784    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"]
Oct 30 20:15:53.329459 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:53.329426    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"]
Oct 30 20:15:54.337902 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.337843    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"]
Oct 30 20:15:54.338435 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338414    4823 cpu_manager.go:414] "containerMap: removing container" podUID="43b70bdc-bb09-4baa-ac3c-dbbefb3adf27" containerName="main"
Oct 30 20:15:54.338435 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338435    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b70bdc-bb09-4baa-ac3c-dbbefb3adf27" containerName="main"
Oct 30 20:15:54.338539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338447    4823 cpu_manager.go:414] "containerMap: removing container" podUID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" containerName="istio-proxy"
Oct 30 20:15:54.338539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338453    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" containerName="istio-proxy"
Oct 30 20:15:54.338539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338462    4823 cpu_manager.go:414] "containerMap: removing container" podUID="ee26e2d8-c730-4c60-97ce-f76495b990a8" containerName="cleanup"
Oct 30 20:15:54.338539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338467    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee26e2d8-c730-4c60-97ce-f76495b990a8" containerName="cleanup"
Oct 30 20:15:54.338539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338476    4823 cpu_manager.go:414] "containerMap: removing container" podUID="ee26e2d8-c730-4c60-97ce-f76495b990a8" containerName="cleanup"
Oct 30 20:15:54.338539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338480    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee26e2d8-c730-4c60-97ce-f76495b990a8" containerName="cleanup"
Oct 30 20:15:54.338539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338488    4823 cpu_manager.go:414] "containerMap: removing container" podUID="326b0731-4c61-494d-9de4-0ad913784cf1" containerName="istio-proxy"
Oct 30 20:15:54.338539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338493    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326b0731-4c61-494d-9de4-0ad913784cf1" containerName="istio-proxy"
Oct 30 20:15:54.338539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338503    4823 cpu_manager.go:414] "containerMap: removing container" podUID="326b0731-4c61-494d-9de4-0ad913784cf1" containerName="istio-proxy"
Oct 30 20:15:54.338539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338511    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326b0731-4c61-494d-9de4-0ad913784cf1" containerName="istio-proxy"
Oct 30 20:15:54.338539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338521    4823 cpu_manager.go:414] "containerMap: removing container" podUID="778e153e-48af-4716-9fb2-2095e02c7bc2" containerName="istio-proxy"
Oct 30 20:15:54.338539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338526    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778e153e-48af-4716-9fb2-2095e02c7bc2" containerName="istio-proxy"
Oct 30 20:15:54.338539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338534    4823 cpu_manager.go:414] "containerMap: removing container" podUID="778e153e-48af-4716-9fb2-2095e02c7bc2" containerName="istio-proxy"
Oct 30 20:15:54.338539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338539    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778e153e-48af-4716-9fb2-2095e02c7bc2" containerName="istio-proxy"
Oct 30 20:15:54.338539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338547    4823 cpu_manager.go:414] "containerMap: removing container" podUID="251fadc1-f1f5-431d-9286-5bd16b99733e" containerName="istio-proxy"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338551    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251fadc1-f1f5-431d-9286-5bd16b99733e" containerName="istio-proxy"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338563    4823 cpu_manager.go:414] "containerMap: removing container" podUID="b3a911c9-4eda-46bd-a390-76b95d5e2c60" containerName="cleanup"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338567    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a911c9-4eda-46bd-a390-76b95d5e2c60" containerName="cleanup"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338578    4823 cpu_manager.go:414] "containerMap: removing container" podUID="b3a911c9-4eda-46bd-a390-76b95d5e2c60" containerName="cleanup"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338582    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a911c9-4eda-46bd-a390-76b95d5e2c60" containerName="cleanup"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338591    4823 cpu_manager.go:414] "containerMap: removing container" podUID="d69555e5-864f-459f-bb32-a8c4fd803b51" containerName="istio-proxy"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338598    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69555e5-864f-459f-bb32-a8c4fd803b51" containerName="istio-proxy"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338608    4823 cpu_manager.go:414] "containerMap: removing container" podUID="b3a911c9-4eda-46bd-a390-76b95d5e2c60" containerName="cleanup"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338612    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a911c9-4eda-46bd-a390-76b95d5e2c60" containerName="cleanup"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338704    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="326b0731-4c61-494d-9de4-0ad913784cf1" containerName="istio-proxy"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338715    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="326b0731-4c61-494d-9de4-0ad913784cf1" containerName="istio-proxy"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338723    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="778e153e-48af-4716-9fb2-2095e02c7bc2" containerName="istio-proxy"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338729    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="778e153e-48af-4716-9fb2-2095e02c7bc2" containerName="istio-proxy"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338738    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="251fadc1-f1f5-431d-9286-5bd16b99733e" containerName="istio-proxy"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338749    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a911c9-4eda-46bd-a390-76b95d5e2c60" containerName="cleanup"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338757    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a911c9-4eda-46bd-a390-76b95d5e2c60" containerName="cleanup"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338765    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69555e5-864f-459f-bb32-a8c4fd803b51" containerName="istio-proxy"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338774    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a911c9-4eda-46bd-a390-76b95d5e2c60" containerName="cleanup"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338784    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" containerName="istio-proxy"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338790    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b70bdc-bb09-4baa-ac3c-dbbefb3adf27" containerName="main"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338797    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" containerName="istio-proxy"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338802    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee26e2d8-c730-4c60-97ce-f76495b990a8" containerName="cleanup"
Oct 30 20:15:54.339224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.338808    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee26e2d8-c730-4c60-97ce-f76495b990a8" containerName="cleanup"
Oct 30 20:15:54.343776 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.343752    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"
Oct 30 20:15:54.355956 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.355929    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:15:54.356088 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.356001    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:15:54.360542 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.360518    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"]
Oct 30 20:15:54.438607 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.438568    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:15:54.438776 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.438614    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:15:54.443385 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.443358    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlbpk\" (UniqueName: \"kubernetes.io/projected/da55fc12-f908-4bbe-9283-7446b1556be9-kube-api-access-xlbpk\") pod \"maas-api-revoke-sub-df9277f6-6qwv7\" (UID: \"da55fc12-f908-4bbe-9283-7446b1556be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"
Oct 30 20:15:54.443525 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.443420    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da55fc12-f908-4bbe-9283-7446b1556be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df9277f6-6qwv7\" (UID: \"da55fc12-f908-4bbe-9283-7446b1556be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"
Oct 30 20:15:54.443525 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.443503    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da55fc12-f908-4bbe-9283-7446b1556be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df9277f6-6qwv7\" (UID: \"da55fc12-f908-4bbe-9283-7446b1556be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"
Oct 30 20:15:54.544909 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.544836    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da55fc12-f908-4bbe-9283-7446b1556be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df9277f6-6qwv7\" (UID: \"da55fc12-f908-4bbe-9283-7446b1556be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"
Oct 30 20:15:54.545099 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.544980    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da55fc12-f908-4bbe-9283-7446b1556be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df9277f6-6qwv7\" (UID: \"da55fc12-f908-4bbe-9283-7446b1556be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"
Oct 30 20:15:54.545168 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.545094    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbpk\" (UniqueName: \"kubernetes.io/projected/da55fc12-f908-4bbe-9283-7446b1556be9-kube-api-access-xlbpk\") pod \"maas-api-revoke-sub-df9277f6-6qwv7\" (UID: \"da55fc12-f908-4bbe-9283-7446b1556be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"
Oct 30 20:15:54.545504 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.545485    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da55fc12-f908-4bbe-9283-7446b1556be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df9277f6-6qwv7\" (UID: \"da55fc12-f908-4bbe-9283-7446b1556be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"
Oct 30 20:15:54.547441 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.547423    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da55fc12-f908-4bbe-9283-7446b1556be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df9277f6-6qwv7\" (UID: \"da55fc12-f908-4bbe-9283-7446b1556be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"
Oct 30 20:15:54.575787 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.575760    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbpk\" (UniqueName: \"kubernetes.io/projected/da55fc12-f908-4bbe-9283-7446b1556be9-kube-api-access-xlbpk\") pod \"maas-api-revoke-sub-df9277f6-6qwv7\" (UID: \"da55fc12-f908-4bbe-9283-7446b1556be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"
Oct 30 20:15:54.660781 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.660757    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"
Oct 30 20:15:54.945679 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:54.945655    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"]
Oct 30 20:15:54.947966 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:15:54.947937    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda55fc12_f908_4bbe_9283_7446b1556be9.slice/crio-58eceb655e99431fe5b1a258e01880f0197fd2aa4723f957d20ecc5061c650ef WatchSource:0}: Error finding container 58eceb655e99431fe5b1a258e01880f0197fd2aa4723f957d20ecc5061c650ef: Status 404 returned error can't find the container with id 58eceb655e99431fe5b1a258e01880f0197fd2aa4723f957d20ecc5061c650ef
Oct 30 20:15:55.896130 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:55.896091    4823 generic.go:356] "Generic (PLEG): container finished" podID="da55fc12-f908-4bbe-9283-7446b1556be9" containerID="2a41cfd9250979b3d35b2b089d8e393f909eaea821dac1425b28ff42a97eeb30" exitCode=0
Oct 30 20:15:55.896537 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:55.896154    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7" event={"ID":"da55fc12-f908-4bbe-9283-7446b1556be9","Type":"ContainerDied","Data":"2a41cfd9250979b3d35b2b089d8e393f909eaea821dac1425b28ff42a97eeb30"}
Oct 30 20:15:55.896537 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:55.896190    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7" event={"ID":"da55fc12-f908-4bbe-9283-7446b1556be9","Type":"ContainerStarted","Data":"58eceb655e99431fe5b1a258e01880f0197fd2aa4723f957d20ecc5061c650ef"}
Oct 30 20:15:57.057668 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:57.057642    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"
Oct 30 20:15:57.170739 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:57.170655    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da55fc12-f908-4bbe-9283-7446b1556be9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da55fc12-f908-4bbe-9283-7446b1556be9-maas-api-xks-ca\") pod \"da55fc12-f908-4bbe-9283-7446b1556be9\" (UID: \"da55fc12-f908-4bbe-9283-7446b1556be9\") "
Oct 30 20:15:57.170930 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:57.170839    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da55fc12-f908-4bbe-9283-7446b1556be9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da55fc12-f908-4bbe-9283-7446b1556be9-maas-api-openshift-ca\") pod \"da55fc12-f908-4bbe-9283-7446b1556be9\" (UID: \"da55fc12-f908-4bbe-9283-7446b1556be9\") "
Oct 30 20:15:57.171021 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:57.170944    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da55fc12-f908-4bbe-9283-7446b1556be9-kube-api-access-xlbpk\" (UniqueName: \"kubernetes.io/projected/da55fc12-f908-4bbe-9283-7446b1556be9-kube-api-access-xlbpk\") pod \"da55fc12-f908-4bbe-9283-7446b1556be9\" (UID: \"da55fc12-f908-4bbe-9283-7446b1556be9\") "
Oct 30 20:15:57.171217 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:57.171193    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da55fc12-f908-4bbe-9283-7446b1556be9-maas-api-openshift-ca" pod "da55fc12-f908-4bbe-9283-7446b1556be9" (UID: "da55fc12-f908-4bbe-9283-7446b1556be9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:57.171344 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:57.171331    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da55fc12-f908-4bbe-9283-7446b1556be9-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:57.172739 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:57.172715    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da55fc12-f908-4bbe-9283-7446b1556be9-maas-api-xks-ca" pod "da55fc12-f908-4bbe-9283-7446b1556be9" (UID: "da55fc12-f908-4bbe-9283-7446b1556be9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:57.172967 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:57.172947    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da55fc12-f908-4bbe-9283-7446b1556be9-kube-api-access-xlbpk" pod "da55fc12-f908-4bbe-9283-7446b1556be9" (UID: "da55fc12-f908-4bbe-9283-7446b1556be9"). InnerVolumeSpecName "kube-api-access-xlbpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:57.271782 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:57.271753    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlbpk\" (UniqueName: \"kubernetes.io/projected/da55fc12-f908-4bbe-9283-7446b1556be9-kube-api-access-xlbpk\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:57.271782 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:57.271778    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da55fc12-f908-4bbe-9283-7446b1556be9-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:57.910280 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:57.910248    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"
Oct 30 20:15:57.910487 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:57.910285    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7" event={"ID":"da55fc12-f908-4bbe-9283-7446b1556be9","Type":"ContainerDied","Data":"58eceb655e99431fe5b1a258e01880f0197fd2aa4723f957d20ecc5061c650ef"}
Oct 30 20:15:57.910487 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:57.910334    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58eceb655e99431fe5b1a258e01880f0197fd2aa4723f957d20ecc5061c650ef"
Oct 30 20:15:58.314498 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:58.314406    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p" podUID="ca8f8f91-8f8b-4f8b-a0d9-3483580a7855" containerName="payload-processing" containerID="cri-o://a0e76d316f44f8576c6a86d71e222be98729fe82c4ff45ec8419f7a11b430c53" gracePeriod=25
Oct 30 20:15:58.329994 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:15:58.329951    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9" podUID="dbb7a21b-3c5f-446b-81ee-732307718f1b" containerName="payload-pre-processing" containerID="cri-o://4259dfbbda23a228d18f5b93f38a56bfc37de94c8167fd8b77b4e4288262c58e" gracePeriod=25
Oct 30 20:16:01.391440 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:01.391399    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p" podUID="ca8f8f91-8f8b-4f8b-a0d9-3483580a7855" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:01.391823 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:01.391410    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9" podUID="dbb7a21b-3c5f-446b-81ee-732307718f1b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:02.013906 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:16:02.013851    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:16:03.025053 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.024992    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"]
Oct 30 20:16:03.028157 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.028131    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1685160c-7l8cb"]
Oct 30 20:16:03.054523 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.054450    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main" containerID="cri-o://520da7347061651d9fe736ebaa3048520d02f137c2332420e130dcce2e108335" gracePeriod=45
Oct 30 20:16:03.064057 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.064025    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" containerID="cri-o://ead3662d68a6479c3abc9c4acc79c953001ed92f9f81e79e738ee9bf75d42536" gracePeriod=45
Oct 30 20:16:03.083186 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.083155    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": EOF"
Oct 30 20:16:03.356567 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.356537    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 20:16:03.373359 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.373330    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 20:16:03.451707 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.451670    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/499b7351-9dea-47b8-8e88-a532f7fab38f-tls-certs\" (UniqueName: \"kubernetes.io/secret/499b7351-9dea-47b8-8e88-a532f7fab38f-tls-certs\") pod \"499b7351-9dea-47b8-8e88-a532f7fab38f\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") "
Oct 30 20:16:03.451928 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.451721    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-home\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-home\") pod \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") "
Oct 30 20:16:03.451928 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.451769    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-dshm\") pod \"499b7351-9dea-47b8-8e88-a532f7fab38f\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") "
Oct 30 20:16:03.451928 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.451824    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-home\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-home\") pod \"499b7351-9dea-47b8-8e88-a532f7fab38f\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") "
Oct 30 20:16:03.451928 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.451852    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-model-cache\") pod \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") "
Oct 30 20:16:03.451928 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.451925    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-tmp-dir\") pod \"499b7351-9dea-47b8-8e88-a532f7fab38f\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") "
Oct 30 20:16:03.452285 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.452077    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-home" pod "499b7351-9dea-47b8-8e88-a532f7fab38f" (UID: "499b7351-9dea-47b8-8e88-a532f7fab38f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.452285 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.452090    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-home" pod "62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" (UID: "62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.452285 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.452182    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-model-cache" pod "62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" (UID: "62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.452285 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.452247    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-tls-certs\" (UniqueName: \"kubernetes.io/secret/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-tls-certs\") pod \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") "
Oct 30 20:16:03.452506 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.452357    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-tmp-dir" pod "499b7351-9dea-47b8-8e88-a532f7fab38f" (UID: "499b7351-9dea-47b8-8e88-a532f7fab38f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.452506 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.452395    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/499b7351-9dea-47b8-8e88-a532f7fab38f-kube-api-access-w6pn8\" (UniqueName: \"kubernetes.io/projected/499b7351-9dea-47b8-8e88-a532f7fab38f-kube-api-access-w6pn8\") pod \"499b7351-9dea-47b8-8e88-a532f7fab38f\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") "
Oct 30 20:16:03.452506 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.452452    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-model-cache\") pod \"499b7351-9dea-47b8-8e88-a532f7fab38f\" (UID: \"499b7351-9dea-47b8-8e88-a532f7fab38f\") "
Oct 30 20:16:03.452506 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.452503    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-dshm\") pod \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") "
Oct 30 20:16:03.452748 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.452728    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-model-cache" pod "499b7351-9dea-47b8-8e88-a532f7fab38f" (UID: "499b7351-9dea-47b8-8e88-a532f7fab38f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.452825 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.452808    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-kube-api-access-jcdm2\" (UniqueName: \"kubernetes.io/projected/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-kube-api-access-jcdm2\") pod \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") "
Oct 30 20:16:03.452930 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.452840    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-tmp-dir\") pod \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\" (UID: \"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718\") "
Oct 30 20:16:03.453234 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.453172    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-tmp-dir" pod "62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" (UID: "62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.453594 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.453570    4823 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-model-cache\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.453704 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.453599    4823 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-tmp-dir\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.453704 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.453615    4823 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-home\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.453704 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.453628    4823 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-home\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.453704 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.453642    4823 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-model-cache\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.453704 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.453655    4823 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-tmp-dir\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.454531 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.454509    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-dshm" pod "499b7351-9dea-47b8-8e88-a532f7fab38f" (UID: "499b7351-9dea-47b8-8e88-a532f7fab38f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.455030 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.455003    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-tls-certs" pod "62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" (UID: "62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:03.455157 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.455137    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/499b7351-9dea-47b8-8e88-a532f7fab38f-tls-certs" pod "499b7351-9dea-47b8-8e88-a532f7fab38f" (UID: "499b7351-9dea-47b8-8e88-a532f7fab38f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:03.455352 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.455332    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499b7351-9dea-47b8-8e88-a532f7fab38f-kube-api-access-w6pn8" pod "499b7351-9dea-47b8-8e88-a532f7fab38f" (UID: "499b7351-9dea-47b8-8e88-a532f7fab38f"). InnerVolumeSpecName "kube-api-access-w6pn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:03.455673 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.455655    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-kube-api-access-jcdm2" pod "62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" (UID: "62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718"). InnerVolumeSpecName "kube-api-access-jcdm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:03.456669 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.456623    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-dshm" pod "62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" (UID: "62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.552297 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.552261    4823 scope.go:122] "RemoveContainer" containerID="520da7347061651d9fe736ebaa3048520d02f137c2332420e130dcce2e108335"
Oct 30 20:16:03.555087 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.555063    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcdm2\" (UniqueName: \"kubernetes.io/projected/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-kube-api-access-jcdm2\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.555192 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.555090    4823 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/499b7351-9dea-47b8-8e88-a532f7fab38f-tls-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.555192 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.555100    4823 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/499b7351-9dea-47b8-8e88-a532f7fab38f-dshm\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.555192 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.555108    4823 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-tls-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.555192 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.555116    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6pn8\" (UniqueName: \"kubernetes.io/projected/499b7351-9dea-47b8-8e88-a532f7fab38f-kube-api-access-w6pn8\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.555192 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.555124    4823 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718-dshm\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.562937 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.562918    4823 scope.go:122] "RemoveContainer" containerID="ead3662d68a6479c3abc9c4acc79c953001ed92f9f81e79e738ee9bf75d42536"
Oct 30 20:16:03.949310 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.949266    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" event={"ID":"499b7351-9dea-47b8-8e88-a532f7fab38f","Type":"ContainerDied","Data":"ead3662d68a6479c3abc9c4acc79c953001ed92f9f81e79e738ee9bf75d42536"}
Oct 30 20:16:03.949477 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.949322    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"
Oct 30 20:16:03.949477 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.949327    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"
Oct 30 20:16:03.949599 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.949326    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz" event={"ID":"499b7351-9dea-47b8-8e88-a532f7fab38f","Type":"ContainerDied","Data":"54c85b2ba71fd5c4eee9bf0bef0db6213e35be9682cb049d9dbf3036f11c281c"}
Oct 30 20:16:03.949649 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.949603    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" event={"ID":"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718","Type":"ContainerDied","Data":"520da7347061651d9fe736ebaa3048520d02f137c2332420e130dcce2e108335"}
Oct 30 20:16:03.949649 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:03.949627    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj" event={"ID":"62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718","Type":"ContainerDied","Data":"9de131df7143f54b41efc2add13f3ee71388f9e9654b9ae9e3b3922828e38ea3"}
Oct 30 20:16:04.003676 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:04.003637    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"]
Oct 30 20:16:04.008909 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:04.008886    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/premium-w3-8a8207-kserve-94d6f88cf-mlnkj"]
Oct 30 20:16:04.023655 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:04.023627    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"]
Oct 30 20:16:04.029123 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:04.029098    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/simulator-w3-8a8207-kserve-864587d478-kj8bz"]
Oct 30 20:16:04.035038 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:04.035012    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"]
Oct 30 20:16:04.039518 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:04.039498    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-889acd9b-9qsdq"]
Oct 30 20:16:04.278275 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:04.278196    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26e56c15-b87d-4b0a-b403-f32809781e5d" path="/var/lib/kubelet/pods/26e56c15-b87d-4b0a-b403-f32809781e5d/volumes"
Oct 30 20:16:04.278627 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:04.278613    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" path="/var/lib/kubelet/pods/499b7351-9dea-47b8-8e88-a532f7fab38f/volumes"
Oct 30 20:16:04.279075 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:04.279056    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" path="/var/lib/kubelet/pods/62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718/volumes"
Oct 30 20:16:04.279800 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:04.279785    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eef12d45-3357-4f21-9d3e-3d073a82cc3b" path="/var/lib/kubelet/pods/eef12d45-3357-4f21-9d3e-3d073a82cc3b/volumes"
Oct 30 20:16:06.635019 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:06.634985    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:16:11.391994 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:11.391954    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p" podUID="ca8f8f91-8f8b-4f8b-a0d9-3483580a7855" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:11.392485 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:11.391958    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9" podUID="dbb7a21b-3c5f-446b-81ee-732307718f1b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:16:14.730853 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:14.730818    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:16:18.452189 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.452154    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"]
Oct 30 20:16:18.452581 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.452371    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz" podUID="faf2ad6b-cfdb-4e4a-99e5-1021675227c2" containerName="istio-proxy" containerID="cri-o://670106df987067313419fb56598633b9fd80dc22f729fcd5915de7a217bf1c97" gracePeriod=30
Oct 30 20:16:18.456235 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.456211    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"]
Oct 30 20:16:18.456448 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.456424    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb" podUID="e473bfd6-107c-4cc1-a9f8-8f8e546b743b" containerName="istio-proxy" containerID="cri-o://27676435341d0f90cb433f0fa7d9007d60ceba3f562c8544bd556ab4c19c3d65" gracePeriod=30
Oct 30 20:16:18.607278 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.607253    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"
Oct 30 20:16:18.647142 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.647121    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"
Oct 30 20:16:18.722303 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.722274    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8f8f91-8f8b-4f8b-a0d9-3483580a7855-kube-api-access-68g6x\" (UniqueName: \"kubernetes.io/projected/ca8f8f91-8f8b-4f8b-a0d9-3483580a7855-kube-api-access-68g6x\") pod \"ca8f8f91-8f8b-4f8b-a0d9-3483580a7855\" (UID: \"ca8f8f91-8f8b-4f8b-a0d9-3483580a7855\") "
Oct 30 20:16:18.722501 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.722351    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca8f8f91-8f8b-4f8b-a0d9-3483580a7855-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8f8f91-8f8b-4f8b-a0d9-3483580a7855-plugins-config-volume\") pod \"ca8f8f91-8f8b-4f8b-a0d9-3483580a7855\" (UID: \"ca8f8f91-8f8b-4f8b-a0d9-3483580a7855\") "
Oct 30 20:16:18.722765 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.722743    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca8f8f91-8f8b-4f8b-a0d9-3483580a7855-plugins-config-volume" pod "ca8f8f91-8f8b-4f8b-a0d9-3483580a7855" (UID: "ca8f8f91-8f8b-4f8b-a0d9-3483580a7855"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:18.724467 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.724444    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8f8f91-8f8b-4f8b-a0d9-3483580a7855-kube-api-access-68g6x" pod "ca8f8f91-8f8b-4f8b-a0d9-3483580a7855" (UID: "ca8f8f91-8f8b-4f8b-a0d9-3483580a7855"). InnerVolumeSpecName "kube-api-access-68g6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:18.823539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.823466    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb7a21b-3c5f-446b-81ee-732307718f1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb7a21b-3c5f-446b-81ee-732307718f1b-plugins-config-volume\") pod \"dbb7a21b-3c5f-446b-81ee-732307718f1b\" (UID: \"dbb7a21b-3c5f-446b-81ee-732307718f1b\") "
Oct 30 20:16:18.823539 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.823533    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb7a21b-3c5f-446b-81ee-732307718f1b-kube-api-access-7nvv9\" (UniqueName: \"kubernetes.io/projected/dbb7a21b-3c5f-446b-81ee-732307718f1b-kube-api-access-7nvv9\") pod \"dbb7a21b-3c5f-446b-81ee-732307718f1b\" (UID: \"dbb7a21b-3c5f-446b-81ee-732307718f1b\") "
Oct 30 20:16:18.823785 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.823770    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68g6x\" (UniqueName: \"kubernetes.io/projected/ca8f8f91-8f8b-4f8b-a0d9-3483580a7855-kube-api-access-68g6x\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:18.823854 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.823777    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb7a21b-3c5f-446b-81ee-732307718f1b-plugins-config-volume" pod "dbb7a21b-3c5f-446b-81ee-732307718f1b" (UID: "dbb7a21b-3c5f-446b-81ee-732307718f1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:18.823854 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.823786    4823 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8f8f91-8f8b-4f8b-a0d9-3483580a7855-plugins-config-volume\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:18.825632 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.825611    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb7a21b-3c5f-446b-81ee-732307718f1b-kube-api-access-7nvv9" pod "dbb7a21b-3c5f-446b-81ee-732307718f1b" (UID: "dbb7a21b-3c5f-446b-81ee-732307718f1b"). InnerVolumeSpecName "kube-api-access-7nvv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:18.924947 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.924909    4823 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb7a21b-3c5f-446b-81ee-732307718f1b-plugins-config-volume\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:18.924947 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:18.924940    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nvv9\" (UniqueName: \"kubernetes.io/projected/dbb7a21b-3c5f-446b-81ee-732307718f1b-kube-api-access-7nvv9\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:19.054121 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.054090    4823 generic.go:356] "Generic (PLEG): container finished" podID="dbb7a21b-3c5f-446b-81ee-732307718f1b" containerID="4259dfbbda23a228d18f5b93f38a56bfc37de94c8167fd8b77b4e4288262c58e" exitCode=0
Oct 30 20:16:19.054333 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.054136    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"
Oct 30 20:16:19.054333 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.054175    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9" event={"ID":"dbb7a21b-3c5f-446b-81ee-732307718f1b","Type":"ContainerDied","Data":"4259dfbbda23a228d18f5b93f38a56bfc37de94c8167fd8b77b4e4288262c58e"}
Oct 30 20:16:19.054333 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.054240    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9" event={"ID":"dbb7a21b-3c5f-446b-81ee-732307718f1b","Type":"ContainerDied","Data":"ac1d839ea6831fed4fa4e51e4de05d96626c1ef2306f31602741a04269e44082"}
Oct 30 20:16:19.054333 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.054271    4823 scope.go:122] "RemoveContainer" containerID="4259dfbbda23a228d18f5b93f38a56bfc37de94c8167fd8b77b4e4288262c58e"
Oct 30 20:16:19.055626 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.055602    4823 generic.go:356] "Generic (PLEG): container finished" podID="ca8f8f91-8f8b-4f8b-a0d9-3483580a7855" containerID="a0e76d316f44f8576c6a86d71e222be98729fe82c4ff45ec8419f7a11b430c53" exitCode=0
Oct 30 20:16:19.055710 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.055643    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"
Oct 30 20:16:19.055710 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.055678    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p" event={"ID":"ca8f8f91-8f8b-4f8b-a0d9-3483580a7855","Type":"ContainerDied","Data":"a0e76d316f44f8576c6a86d71e222be98729fe82c4ff45ec8419f7a11b430c53"}
Oct 30 20:16:19.055788 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.055728    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p" event={"ID":"ca8f8f91-8f8b-4f8b-a0d9-3483580a7855","Type":"ContainerDied","Data":"64a9320ecb12f1c6310b80741f99423689aaee97a72465b8932e788e3614c1f6"}
Oct 30 20:16:19.065474 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.065455    4823 scope.go:122] "RemoveContainer" containerID="4259dfbbda23a228d18f5b93f38a56bfc37de94c8167fd8b77b4e4288262c58e"
Oct 30 20:16:19.065811 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:16:19.065785    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4259dfbbda23a228d18f5b93f38a56bfc37de94c8167fd8b77b4e4288262c58e\": container with ID starting with 4259dfbbda23a228d18f5b93f38a56bfc37de94c8167fd8b77b4e4288262c58e not found: ID does not exist" containerID="4259dfbbda23a228d18f5b93f38a56bfc37de94c8167fd8b77b4e4288262c58e"
Oct 30 20:16:19.065913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.065819    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4259dfbbda23a228d18f5b93f38a56bfc37de94c8167fd8b77b4e4288262c58e"} err="failed to get container status \"4259dfbbda23a228d18f5b93f38a56bfc37de94c8167fd8b77b4e4288262c58e\": rpc error: code = NotFound desc = could not find container \"4259dfbbda23a228d18f5b93f38a56bfc37de94c8167fd8b77b4e4288262c58e\": container with ID starting with 4259dfbbda23a228d18f5b93f38a56bfc37de94c8167fd8b77b4e4288262c58e not found: ID does not exist"
Oct 30 20:16:19.065913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.065836    4823 scope.go:122] "RemoveContainer" containerID="a0e76d316f44f8576c6a86d71e222be98729fe82c4ff45ec8419f7a11b430c53"
Oct 30 20:16:19.075046 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.075028    4823 scope.go:122] "RemoveContainer" containerID="a0e76d316f44f8576c6a86d71e222be98729fe82c4ff45ec8419f7a11b430c53"
Oct 30 20:16:19.075294 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:16:19.075273    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e76d316f44f8576c6a86d71e222be98729fe82c4ff45ec8419f7a11b430c53\": container with ID starting with a0e76d316f44f8576c6a86d71e222be98729fe82c4ff45ec8419f7a11b430c53 not found: ID does not exist" containerID="a0e76d316f44f8576c6a86d71e222be98729fe82c4ff45ec8419f7a11b430c53"
Oct 30 20:16:19.075352 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.075301    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e76d316f44f8576c6a86d71e222be98729fe82c4ff45ec8419f7a11b430c53"} err="failed to get container status \"a0e76d316f44f8576c6a86d71e222be98729fe82c4ff45ec8419f7a11b430c53\": rpc error: code = NotFound desc = could not find container \"a0e76d316f44f8576c6a86d71e222be98729fe82c4ff45ec8419f7a11b430c53\": container with ID starting with a0e76d316f44f8576c6a86d71e222be98729fe82c4ff45ec8419f7a11b430c53 not found: ID does not exist"
Oct 30 20:16:19.115033 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.115010    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"]
Oct 30 20:16:19.124030 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.124008    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8a8207-687985667f-fpmf9"]
Oct 30 20:16:19.158114 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.158083    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"]
Oct 30 20:16:19.162576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.162549    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8a8207-869fddd464-7ff6p"]
Oct 30 20:16:19.676062 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.676025    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:16:19.676475 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.676103    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb" podUID="e473bfd6-107c-4cc1-a9f8-8f8e546b743b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:16:19.944136 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.944048    4823 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:16:19.944136 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:19.944109    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz" podUID="faf2ad6b-cfdb-4e4a-99e5-1021675227c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:16:20.279467 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:20.279391    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca8f8f91-8f8b-4f8b-a0d9-3483580a7855" path="/var/lib/kubelet/pods/ca8f8f91-8f8b-4f8b-a0d9-3483580a7855/volumes"
Oct 30 20:16:20.279802 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:20.279789    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb7a21b-3c5f-446b-81ee-732307718f1b" path="/var/lib/kubelet/pods/dbb7a21b-3c5f-446b-81ee-732307718f1b/volumes"
Oct 30 20:16:23.760673 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.760608    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 20:16:23.919064 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.919028    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-token\" (UniqueName: \"kubernetes.io/projected/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-token\") pod \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") "
Oct 30 20:16:23.919229 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.919098    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-workload-socket\") pod \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") "
Oct 30 20:16:23.919229 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.919150    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-podinfo\") pod \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") "
Oct 30 20:16:23.919229 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.919180    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-data\") pod \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") "
Oct 30 20:16:23.919380 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.919233    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-kube-api-access-csj8n\" (UniqueName: \"kubernetes.io/projected/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-kube-api-access-csj8n\") pod \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") "
Oct 30 20:16:23.919380 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.919272    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istiod-ca-cert\") pod \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") "
Oct 30 20:16:23.919380 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.919320    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-credential-socket\") pod \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") "
Oct 30 20:16:23.919380 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.919354    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-envoy\") pod \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") "
Oct 30 20:16:23.919505 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.919394    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-workload-certs\") pod \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\" (UID: \"faf2ad6b-cfdb-4e4a-99e5-1021675227c2\") "
Oct 30 20:16:23.920393 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.919789    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-data" pod "faf2ad6b-cfdb-4e4a-99e5-1021675227c2" (UID: "faf2ad6b-cfdb-4e4a-99e5-1021675227c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:23.920393 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.919807    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-credential-socket" pod "faf2ad6b-cfdb-4e4a-99e5-1021675227c2" (UID: "faf2ad6b-cfdb-4e4a-99e5-1021675227c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:23.920393 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.919904    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:23.920393 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.919928    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:23.920393 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.920067    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-workload-certs" pod "faf2ad6b-cfdb-4e4a-99e5-1021675227c2" (UID: "faf2ad6b-cfdb-4e4a-99e5-1021675227c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:23.920393 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.920163    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istiod-ca-cert" pod "faf2ad6b-cfdb-4e4a-99e5-1021675227c2" (UID: "faf2ad6b-cfdb-4e4a-99e5-1021675227c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:23.920393 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.920353    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-workload-socket" pod "faf2ad6b-cfdb-4e4a-99e5-1021675227c2" (UID: "faf2ad6b-cfdb-4e4a-99e5-1021675227c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:23.922285 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.922224    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-kube-api-access-csj8n" pod "faf2ad6b-cfdb-4e4a-99e5-1021675227c2" (UID: "faf2ad6b-cfdb-4e4a-99e5-1021675227c2"). InnerVolumeSpecName "kube-api-access-csj8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:23.922523 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.922490    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-envoy" pod "faf2ad6b-cfdb-4e4a-99e5-1021675227c2" (UID: "faf2ad6b-cfdb-4e4a-99e5-1021675227c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:23.922650 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.922594    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-podinfo" pod "faf2ad6b-cfdb-4e4a-99e5-1021675227c2" (UID: "faf2ad6b-cfdb-4e4a-99e5-1021675227c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:16:23.923216 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:23.923186    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-token" pod "faf2ad6b-cfdb-4e4a-99e5-1021675227c2" (UID: "faf2ad6b-cfdb-4e4a-99e5-1021675227c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:24.020628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.020596    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.020628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.020629    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.020794 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.020640    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csj8n\" (UniqueName: \"kubernetes.io/projected/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-kube-api-access-csj8n\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.020794 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.020651    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.020794 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.020660    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.020794 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.020669    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.020794 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.020677    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf2ad6b-cfdb-4e4a-99e5-1021675227c2-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.115462 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.115202    4823 generic.go:356] "Generic (PLEG): container finished" podID="e473bfd6-107c-4cc1-a9f8-8f8e546b743b" containerID="27676435341d0f90cb433f0fa7d9007d60ceba3f562c8544bd556ab4c19c3d65" exitCode=0
Oct 30 20:16:24.115462 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.115348    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb" event={"ID":"e473bfd6-107c-4cc1-a9f8-8f8e546b743b","Type":"ContainerDied","Data":"27676435341d0f90cb433f0fa7d9007d60ceba3f562c8544bd556ab4c19c3d65"}
Oct 30 20:16:24.119894 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.117565    4823 generic.go:356] "Generic (PLEG): container finished" podID="faf2ad6b-cfdb-4e4a-99e5-1021675227c2" containerID="670106df987067313419fb56598633b9fd80dc22f729fcd5915de7a217bf1c97" exitCode=0
Oct 30 20:16:24.119894 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.117769    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz" event={"ID":"faf2ad6b-cfdb-4e4a-99e5-1021675227c2","Type":"ContainerDied","Data":"670106df987067313419fb56598633b9fd80dc22f729fcd5915de7a217bf1c97"}
Oct 30 20:16:24.119894 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.117813    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz" event={"ID":"faf2ad6b-cfdb-4e4a-99e5-1021675227c2","Type":"ContainerDied","Data":"61d16eea8dfe16968d4a6fe41ea883810e9cf4c6954e0746cdfaaf3d18fcdcad"}
Oct 30 20:16:24.119894 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.117843    4823 scope.go:122] "RemoveContainer" containerID="670106df987067313419fb56598633b9fd80dc22f729fcd5915de7a217bf1c97"
Oct 30 20:16:24.119894 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.118015    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"
Oct 30 20:16:24.134924 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.134891    4823 scope.go:122] "RemoveContainer" containerID="670106df987067313419fb56598633b9fd80dc22f729fcd5915de7a217bf1c97"
Oct 30 20:16:24.135350 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:16:24.135319    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670106df987067313419fb56598633b9fd80dc22f729fcd5915de7a217bf1c97\": container with ID starting with 670106df987067313419fb56598633b9fd80dc22f729fcd5915de7a217bf1c97 not found: ID does not exist" containerID="670106df987067313419fb56598633b9fd80dc22f729fcd5915de7a217bf1c97"
Oct 30 20:16:24.135444 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.135362    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670106df987067313419fb56598633b9fd80dc22f729fcd5915de7a217bf1c97"} err="failed to get container status \"670106df987067313419fb56598633b9fd80dc22f729fcd5915de7a217bf1c97\": rpc error: code = NotFound desc = could not find container \"670106df987067313419fb56598633b9fd80dc22f729fcd5915de7a217bf1c97\": container with ID starting with 670106df987067313419fb56598633b9fd80dc22f729fcd5915de7a217bf1c97 not found: ID does not exist"
Oct 30 20:16:24.149923 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.149894    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"]
Oct 30 20:16:24.159993 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.159965    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 20:16:24.168827 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.168799    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-stphz"]
Oct 30 20:16:24.222226 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.222179    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-envoy\") pod \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") "
Oct 30 20:16:24.222829 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.222808    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-kube-api-access-9sdm5\" (UniqueName: \"kubernetes.io/projected/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-kube-api-access-9sdm5\") pod \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") "
Oct 30 20:16:24.223060 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.223045    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-data\") pod \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") "
Oct 30 20:16:24.223213 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.223196    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istiod-ca-cert\") pod \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") "
Oct 30 20:16:24.223706 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.223692    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-credential-socket\") pod \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") "
Oct 30 20:16:24.223826 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.223816    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-workload-socket\") pod \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") "
Oct 30 20:16:24.223950 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.223936    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-workload-certs\") pod \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") "
Oct 30 20:16:24.224424 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.224404    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-podinfo\") pod \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") "
Oct 30 20:16:24.224771 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.224739    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-token\" (UniqueName: \"kubernetes.io/projected/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-token\") pod \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\" (UID: \"e473bfd6-107c-4cc1-a9f8-8f8e546b743b\") "
Oct 30 20:16:24.225091 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.223424    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-data" pod "e473bfd6-107c-4cc1-a9f8-8f8e546b743b" (UID: "e473bfd6-107c-4cc1-a9f8-8f8e546b743b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:24.225189 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.223623    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istiod-ca-cert" pod "e473bfd6-107c-4cc1-a9f8-8f8e546b743b" (UID: "e473bfd6-107c-4cc1-a9f8-8f8e546b743b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:24.225189 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.224138    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-credential-socket" pod "e473bfd6-107c-4cc1-a9f8-8f8e546b743b" (UID: "e473bfd6-107c-4cc1-a9f8-8f8e546b743b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:24.225189 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.224314    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-workload-certs" pod "e473bfd6-107c-4cc1-a9f8-8f8e546b743b" (UID: "e473bfd6-107c-4cc1-a9f8-8f8e546b743b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:24.225189 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.224344    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-workload-socket" pod "e473bfd6-107c-4cc1-a9f8-8f8e546b743b" (UID: "e473bfd6-107c-4cc1-a9f8-8f8e546b743b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:24.225400 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.225270    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-kube-api-access-9sdm5" pod "e473bfd6-107c-4cc1-a9f8-8f8e546b743b" (UID: "e473bfd6-107c-4cc1-a9f8-8f8e546b743b"). InnerVolumeSpecName "kube-api-access-9sdm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:24.225514 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.225500    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-envoy" pod "e473bfd6-107c-4cc1-a9f8-8f8e546b743b" (UID: "e473bfd6-107c-4cc1-a9f8-8f8e546b743b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:24.225850 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.225839    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.226069 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.225910    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sdm5\" (UniqueName: \"kubernetes.io/projected/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-kube-api-access-9sdm5\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.226069 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.225928    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.226069 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.225942    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.226069 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.225955    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.226069 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.225998    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.226069 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.226012    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.226391 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.226363    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-podinfo" pod "e473bfd6-107c-4cc1-a9f8-8f8e546b743b" (UID: "e473bfd6-107c-4cc1-a9f8-8f8e546b743b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:16:24.226529 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.226511    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-token" pod "e473bfd6-107c-4cc1-a9f8-8f8e546b743b" (UID: "e473bfd6-107c-4cc1-a9f8-8f8e546b743b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:24.285366 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.285287    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faf2ad6b-cfdb-4e4a-99e5-1021675227c2" path="/var/lib/kubelet/pods/faf2ad6b-cfdb-4e4a-99e5-1021675227c2/volumes"
Oct 30 20:16:24.327362 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.327327    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:24.327591 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:24.327567    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e473bfd6-107c-4cc1-a9f8-8f8e546b743b-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:25.126311 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:25.126279    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"
Oct 30 20:16:25.126311 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:25.126292    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb" event={"ID":"e473bfd6-107c-4cc1-a9f8-8f8e546b743b","Type":"ContainerDied","Data":"96701ee7a298a03f516a8d273fd7ee2869e8e977a913c45d1c37771fec37a9f1"}
Oct 30 20:16:25.126899 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:25.126359    4823 scope.go:122] "RemoveContainer" containerID="27676435341d0f90cb433f0fa7d9007d60ceba3f562c8544bd556ab4c19c3d65"
Oct 30 20:16:25.158575 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:25.158550    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"]
Oct 30 20:16:25.161926 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:25.161903    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-zqxwb"]
Oct 30 20:16:26.277958 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:26.277922    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e473bfd6-107c-4cc1-a9f8-8f8e546b743b" path="/var/lib/kubelet/pods/e473bfd6-107c-4cc1-a9f8-8f8e546b743b/volumes"
Oct 30 20:16:32.013670 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:16:32.013634    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:16:42.757655 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.757622    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"]
Oct 30 20:16:42.758112 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758097    4823 cpu_manager.go:414] "containerMap: removing container" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main"
Oct 30 20:16:42.758159 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758113    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main"
Oct 30 20:16:42.758159 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758129    4823 cpu_manager.go:414] "containerMap: removing container" podUID="e473bfd6-107c-4cc1-a9f8-8f8e546b743b" containerName="istio-proxy"
Oct 30 20:16:42.758159 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758134    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e473bfd6-107c-4cc1-a9f8-8f8e546b743b" containerName="istio-proxy"
Oct 30 20:16:42.758159 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758147    4823 cpu_manager.go:414] "containerMap: removing container" podUID="ca8f8f91-8f8b-4f8b-a0d9-3483580a7855" containerName="payload-processing"
Oct 30 20:16:42.758159 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758153    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca8f8f91-8f8b-4f8b-a0d9-3483580a7855" containerName="payload-processing"
Oct 30 20:16:42.758159 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758160    4823 cpu_manager.go:414] "containerMap: removing container" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main"
Oct 30 20:16:42.758357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758166    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main"
Oct 30 20:16:42.758357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758174    4823 cpu_manager.go:414] "containerMap: removing container" podUID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" containerName="istio-proxy"
Oct 30 20:16:42.758357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758186    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda5b0d8-6d21-4ff4-a860-d0f63f912ee0" containerName="istio-proxy"
Oct 30 20:16:42.758357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758201    4823 cpu_manager.go:414] "containerMap: removing container" podUID="faf2ad6b-cfdb-4e4a-99e5-1021675227c2" containerName="istio-proxy"
Oct 30 20:16:42.758357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758206    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf2ad6b-cfdb-4e4a-99e5-1021675227c2" containerName="istio-proxy"
Oct 30 20:16:42.758357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758215    4823 cpu_manager.go:414] "containerMap: removing container" podUID="da55fc12-f908-4bbe-9283-7446b1556be9" containerName="revoke-keys"
Oct 30 20:16:42.758357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758219    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da55fc12-f908-4bbe-9283-7446b1556be9" containerName="revoke-keys"
Oct 30 20:16:42.758357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758237    4823 cpu_manager.go:414] "containerMap: removing container" podUID="dbb7a21b-3c5f-446b-81ee-732307718f1b" containerName="payload-pre-processing"
Oct 30 20:16:42.758357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758242    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb7a21b-3c5f-446b-81ee-732307718f1b" containerName="payload-pre-processing"
Oct 30 20:16:42.758357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758311    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca8f8f91-8f8b-4f8b-a0d9-3483580a7855" containerName="payload-processing"
Oct 30 20:16:42.758357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758319    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c3dcdc-3ec3-4a36-b7ac-2543ff2c1718" containerName="main"
Oct 30 20:16:42.758357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758327    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf2ad6b-cfdb-4e4a-99e5-1021675227c2" containerName="istio-proxy"
Oct 30 20:16:42.758357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758333    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="da55fc12-f908-4bbe-9283-7446b1556be9" containerName="revoke-keys"
Oct 30 20:16:42.758357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758340    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb7a21b-3c5f-446b-81ee-732307718f1b" containerName="payload-pre-processing"
Oct 30 20:16:42.758357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758346    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="499b7351-9dea-47b8-8e88-a532f7fab38f" containerName="main"
Oct 30 20:16:42.758357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.758351    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="e473bfd6-107c-4cc1-a9f8-8f8e546b743b" containerName="istio-proxy"
Oct 30 20:16:42.761736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.761714    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"
Oct 30 20:16:42.769101 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.769073    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:16:42.769796 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.769772    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:16:42.769958 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.769781    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:16:42.769958 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.769805    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:16:42.773631 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.773606    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"]
Oct 30 20:16:42.918913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.918857    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbml4\" (UniqueName: \"kubernetes.io/projected/5e15244c-6697-44f9-b284-5538edb35196-kube-api-access-qbml4\") pod \"maas-api-revoke-sub-85237e45-lf8r9\" (UID: \"5e15244c-6697-44f9-b284-5538edb35196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"
Oct 30 20:16:42.919100 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.918962    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e15244c-6697-44f9-b284-5538edb35196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-lf8r9\" (UID: \"5e15244c-6697-44f9-b284-5538edb35196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"
Oct 30 20:16:42.919100 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:42.919082    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e15244c-6697-44f9-b284-5538edb35196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-lf8r9\" (UID: \"5e15244c-6697-44f9-b284-5538edb35196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"
Oct 30 20:16:43.019656 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:43.019577    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbml4\" (UniqueName: \"kubernetes.io/projected/5e15244c-6697-44f9-b284-5538edb35196-kube-api-access-qbml4\") pod \"maas-api-revoke-sub-85237e45-lf8r9\" (UID: \"5e15244c-6697-44f9-b284-5538edb35196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"
Oct 30 20:16:43.019656 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:43.019624    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e15244c-6697-44f9-b284-5538edb35196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-lf8r9\" (UID: \"5e15244c-6697-44f9-b284-5538edb35196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"
Oct 30 20:16:43.019844 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:43.019752    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e15244c-6697-44f9-b284-5538edb35196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-lf8r9\" (UID: \"5e15244c-6697-44f9-b284-5538edb35196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"
Oct 30 20:16:43.020499 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:43.020471    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e15244c-6697-44f9-b284-5538edb35196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-lf8r9\" (UID: \"5e15244c-6697-44f9-b284-5538edb35196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"
Oct 30 20:16:43.022068 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:43.022049    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e15244c-6697-44f9-b284-5538edb35196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-lf8r9\" (UID: \"5e15244c-6697-44f9-b284-5538edb35196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"
Oct 30 20:16:43.031982 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:43.031948    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbml4\" (UniqueName: \"kubernetes.io/projected/5e15244c-6697-44f9-b284-5538edb35196-kube-api-access-qbml4\") pod \"maas-api-revoke-sub-85237e45-lf8r9\" (UID: \"5e15244c-6697-44f9-b284-5538edb35196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"
Oct 30 20:16:43.076354 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:43.076330    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"
Oct 30 20:16:43.356903 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:43.356875    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"]
Oct 30 20:16:43.359172 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:16:43.359141    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e15244c_6697_44f9_b284_5538edb35196.slice/crio-a896e6a2868997f1c9c6b3404884e1ff7293aa1c707870876b0ef7e8d6a5bb7a WatchSource:0}: Error finding container a896e6a2868997f1c9c6b3404884e1ff7293aa1c707870876b0ef7e8d6a5bb7a: Status 404 returned error can't find the container with id a896e6a2868997f1c9c6b3404884e1ff7293aa1c707870876b0ef7e8d6a5bb7a
Oct 30 20:16:44.027823 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:44.027728    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"]
Oct 30 20:16:44.030935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:44.030908    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbfb04aa-qd7xt"]
Oct 30 20:16:44.255737 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:44.255702    4823 generic.go:356] "Generic (PLEG): container finished" podID="5e15244c-6697-44f9-b284-5538edb35196" containerID="07732c33036cf9edb7040f6ee86eb7b032b7776190e95dd3c657b2d98c9ff4a0" exitCode=0
Oct 30 20:16:44.255950 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:44.255747    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9" event={"ID":"5e15244c-6697-44f9-b284-5538edb35196","Type":"ContainerDied","Data":"07732c33036cf9edb7040f6ee86eb7b032b7776190e95dd3c657b2d98c9ff4a0"}
Oct 30 20:16:44.255950 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:44.255783    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9" event={"ID":"5e15244c-6697-44f9-b284-5538edb35196","Type":"ContainerStarted","Data":"a896e6a2868997f1c9c6b3404884e1ff7293aa1c707870876b0ef7e8d6a5bb7a"}
Oct 30 20:16:44.280014 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:44.279941    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4feaeb6-8155-40f0-b659-acc2d1953d4f" path="/var/lib/kubelet/pods/f4feaeb6-8155-40f0-b659-acc2d1953d4f/volumes"
Oct 30 20:16:45.026457 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:45.026421    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"]
Oct 30 20:16:45.031831 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:45.031807    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3623be56-85h9b"]
Oct 30 20:16:45.416104 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:45.416079    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"
Oct 30 20:16:45.545343 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:45.545309    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e15244c-6697-44f9-b284-5538edb35196-kube-api-access-qbml4\" (UniqueName: \"kubernetes.io/projected/5e15244c-6697-44f9-b284-5538edb35196-kube-api-access-qbml4\") pod \"5e15244c-6697-44f9-b284-5538edb35196\" (UID: \"5e15244c-6697-44f9-b284-5538edb35196\") "
Oct 30 20:16:45.545552 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:45.545424    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e15244c-6697-44f9-b284-5538edb35196-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e15244c-6697-44f9-b284-5538edb35196-maas-api-xks-ca\") pod \"5e15244c-6697-44f9-b284-5538edb35196\" (UID: \"5e15244c-6697-44f9-b284-5538edb35196\") "
Oct 30 20:16:45.545552 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:45.545497    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e15244c-6697-44f9-b284-5538edb35196-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e15244c-6697-44f9-b284-5538edb35196-maas-api-openshift-ca\") pod \"5e15244c-6697-44f9-b284-5538edb35196\" (UID: \"5e15244c-6697-44f9-b284-5538edb35196\") "
Oct 30 20:16:45.545957 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:45.545920    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e15244c-6697-44f9-b284-5538edb35196-maas-api-openshift-ca" pod "5e15244c-6697-44f9-b284-5538edb35196" (UID: "5e15244c-6697-44f9-b284-5538edb35196"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:45.547513 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:45.547483    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e15244c-6697-44f9-b284-5538edb35196-kube-api-access-qbml4" pod "5e15244c-6697-44f9-b284-5538edb35196" (UID: "5e15244c-6697-44f9-b284-5538edb35196"). InnerVolumeSpecName "kube-api-access-qbml4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:45.547617 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:45.547596    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e15244c-6697-44f9-b284-5538edb35196-maas-api-xks-ca" pod "5e15244c-6697-44f9-b284-5538edb35196" (UID: "5e15244c-6697-44f9-b284-5538edb35196"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:45.647135 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:45.647104    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e15244c-6697-44f9-b284-5538edb35196-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:45.647135 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:45.647134    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e15244c-6697-44f9-b284-5538edb35196-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:45.647353 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:45.647145    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbml4\" (UniqueName: \"kubernetes.io/projected/5e15244c-6697-44f9-b284-5538edb35196-kube-api-access-qbml4\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:46.269396 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:46.269365    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"
Oct 30 20:16:46.269836 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:46.269391    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9" event={"ID":"5e15244c-6697-44f9-b284-5538edb35196","Type":"ContainerDied","Data":"a896e6a2868997f1c9c6b3404884e1ff7293aa1c707870876b0ef7e8d6a5bb7a"}
Oct 30 20:16:46.269836 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:46.269475    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a896e6a2868997f1c9c6b3404884e1ff7293aa1c707870876b0ef7e8d6a5bb7a"
Oct 30 20:16:46.277131 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:46.277106    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad764bc6-95ee-432f-940b-c56c4b2cea63" path="/var/lib/kubelet/pods/ad764bc6-95ee-432f-940b-c56c4b2cea63/volumes"
Oct 30 20:16:47.728222 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.728183    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"]
Oct 30 20:16:47.728874 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.728839    4823 cpu_manager.go:414] "containerMap: removing container" podUID="5e15244c-6697-44f9-b284-5538edb35196" containerName="revoke-keys"
Oct 30 20:16:47.729000 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.728877    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e15244c-6697-44f9-b284-5538edb35196" containerName="revoke-keys"
Oct 30 20:16:47.729104 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.729089    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e15244c-6697-44f9-b284-5538edb35196" containerName="revoke-keys"
Oct 30 20:16:47.732718 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.732699    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.756077 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.756048    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"]
Oct 30 20:16:47.869146 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.869108    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.869337 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.869153    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.869337 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.869174    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.869337 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.869294    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8574d565-f870-4374-97ca-7e5bee83f2a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.869337 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.869326    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.869483 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.869404    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.869523 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.869481    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.869558 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.869538    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.869596 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.869565    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqlwk\" (UniqueName: \"kubernetes.io/projected/8574d565-f870-4374-97ca-7e5bee83f2a3-kube-api-access-sqlwk\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.970327 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.970290    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.970327 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.970340    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.970593 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.970360    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqlwk\" (UniqueName: \"kubernetes.io/projected/8574d565-f870-4374-97ca-7e5bee83f2a3-kube-api-access-sqlwk\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.970593 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.970418    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.970593 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.970444    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.970593 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.970459    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.970593 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.970499    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8574d565-f870-4374-97ca-7e5bee83f2a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.970593 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.970573    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.970993 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.970620    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.970993 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.970939    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.970993 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.970976    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.971162 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.971139    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.971234 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.971197    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.971441 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.971414    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8574d565-f870-4374-97ca-7e5bee83f2a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.972772 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.972753    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.973135 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.973114    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.982839 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.982785    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:47.982839 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:47.982807    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqlwk\" (UniqueName: \"kubernetes.io/projected/8574d565-f870-4374-97ca-7e5bee83f2a3-kube-api-access-sqlwk\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8krz9\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:48.049675 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:48.049641    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:48.335725 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:48.335698    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"]
Oct 30 20:16:48.337740 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:16:48.337707    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8574d565_f870_4374_97ca_7e5bee83f2a3.slice/crio-ef323fbcbc9b6ce3c561343baa238671a671278131f99f1229d32af4c65d31a2 WatchSource:0}: Error finding container ef323fbcbc9b6ce3c561343baa238671a671278131f99f1229d32af4c65d31a2: Status 404 returned error can't find the container with id ef323fbcbc9b6ce3c561343baa238671a671278131f99f1229d32af4c65d31a2
Oct 30 20:16:48.340600 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:48.340560    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:48.340723 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:48.340651    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:48.340723 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:48.340695    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:16:49.288536 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:49.288493    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9" event={"ID":"8574d565-f870-4374-97ca-7e5bee83f2a3","Type":"ContainerStarted","Data":"6a71ab5be62dc1dea92cb9ce19137c8d599477f8030d69a6a6ee544cc96ca451"}
Oct 30 20:16:49.289034 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:49.288552    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9" event={"ID":"8574d565-f870-4374-97ca-7e5bee83f2a3","Type":"ContainerStarted","Data":"ef323fbcbc9b6ce3c561343baa238671a671278131f99f1229d32af4c65d31a2"}
Oct 30 20:16:49.314504 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:49.314453    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9" podStartSLOduration=2.314438074 podStartE2EDuration="2.314438074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:49.311899962 +0000 UTC m=+2809.204514287" watchObservedRunningTime="2026-10-30 20:16:49.314438074 +0000 UTC m=+2809.207052398"
Oct 30 20:16:50.050737 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:50.050703    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:50.077077 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:50.077034    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8krz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Oct 30 20:16:50.077259 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:50.077108    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9" podUID="8574d565-f870-4374-97ca-7e5bee83f2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Oct 30 20:16:51.050046 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:51.050013    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8krz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Oct 30 20:16:51.050446 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:51.050062    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9" podUID="8574d565-f870-4374-97ca-7e5bee83f2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Oct 30 20:16:52.050033 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:52.049997    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8krz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Oct 30 20:16:52.050227 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:52.050053    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9" podUID="8574d565-f870-4374-97ca-7e5bee83f2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Oct 30 20:16:53.050443 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:53.050403    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8krz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Oct 30 20:16:53.050917 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:53.050458    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9" podUID="8574d565-f870-4374-97ca-7e5bee83f2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Oct 30 20:16:54.050209 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:54.050161    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8krz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Oct 30 20:16:54.050399 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:54.050223    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9" podUID="8574d565-f870-4374-97ca-7e5bee83f2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Oct 30 20:16:55.050993 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:55.050953    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8krz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Oct 30 20:16:55.051517 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:55.051012    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9" podUID="8574d565-f870-4374-97ca-7e5bee83f2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Oct 30 20:16:56.597783 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:56.597737    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:56.598294 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:56.598245    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:56.598851 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:56.598834    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:16:59.029687 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:59.029650    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"]
Oct 30 20:16:59.040922 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:59.040893    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"]
Oct 30 20:16:59.043597 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:59.043569    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82225a9f-9fqmh"]
Oct 30 20:16:59.050771 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:16:59.050743    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56935a3f-8vrjn"]
Oct 30 20:17:00.277961 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:00.277925    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c6e7d8f-a41b-4609-a6b4-e0642890b4b4" path="/var/lib/kubelet/pods/3c6e7d8f-a41b-4609-a6b4-e0642890b4b4/volumes"
Oct 30 20:17:00.278462 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:00.278437    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db51eeaa-23d4-4ade-9e6e-6c59fae623c0" path="/var/lib/kubelet/pods/db51eeaa-23d4-4ade-9e6e-6c59fae623c0/volumes"
Oct 30 20:17:02.014337 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:17:02.014300    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:17:03.628264 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:03.628224    4823 scope.go:122] "RemoveContainer" containerID="01eb318c412caf5bed774300b3cdaa9adc16f80e8f65128b0be28c602ee120ef"
Oct 30 20:17:03.637837 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:03.637815    4823 scope.go:122] "RemoveContainer" containerID="24ac747c30a5d0219a4e15b2c9e1e4ab9b34a40a64630ed0722c05a0fe34f451"
Oct 30 20:17:32.014399 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:17:32.014317    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:17:32.394709 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:32.394673    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:17:36.031288 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:36.031248    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"]
Oct 30 20:17:36.037103 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:36.037068    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03784eb-84ppm"]
Oct 30 20:17:36.278246 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:36.278206    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432a6a67-fb8e-46da-aa1d-61aa0f4721a1" path="/var/lib/kubelet/pods/432a6a67-fb8e-46da-aa1d-61aa0f4721a1/volumes"
Oct 30 20:17:49.060943 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:49.060905    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"]
Oct 30 20:17:49.075702 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:49.075670    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"]
Oct 30 20:17:49.091888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:49.091829    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaac20b8-5645z"]
Oct 30 20:17:49.094652 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:49.094622    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-780b2274-swx9p"]
Oct 30 20:17:50.279012 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:50.278969    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c721c44-53fa-40b8-8684-785a1cc884f3" path="/var/lib/kubelet/pods/0c721c44-53fa-40b8-8684-785a1cc884f3/volumes"
Oct 30 20:17:50.279485 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:50.279463    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="697e7d67-0ada-4ce2-968d-fb52077793bc" path="/var/lib/kubelet/pods/697e7d67-0ada-4ce2-968d-fb52077793bc/volumes"
Oct 30 20:17:50.893129 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:50.893100    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:17:56.433259 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:56.433222    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:17:57.037603 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:57.037565    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"]
Oct 30 20:17:57.040270 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:57.040243    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f14ab15-9zjfz"]
Oct 30 20:17:57.045898 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:57.045873    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"]
Oct 30 20:17:57.049196 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:57.049173    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fa7f545-kfld5"]
Oct 30 20:17:58.278382 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:58.278343    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7081733e-6a21-4108-bd76-40c7e4d98ea6" path="/var/lib/kubelet/pods/7081733e-6a21-4108-bd76-40c7e4d98ea6/volumes"
Oct 30 20:17:58.278771 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:17:58.278759    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6b6589b-3575-4241-820e-d17634751dac" path="/var/lib/kubelet/pods/c6b6589b-3575-4241-820e-d17634751dac/volumes"
Oct 30 20:18:02.014045 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:18:02.014004    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:18:03.686981 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:03.686952    4823 scope.go:122] "RemoveContainer" containerID="6ef42408679cd75cee44534f7f85ff645ce7b9540012a940706d56be6ad2aa63"
Oct 30 20:18:03.696582 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:03.696560    4823 scope.go:122] "RemoveContainer" containerID="2339a8896d9265976052df5a6f28d7e9ddc53d71e0ceff8acb080b3e7c8b7c1a"
Oct 30 20:18:03.705162 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:03.705132    4823 scope.go:122] "RemoveContainer" containerID="c55ac7a0808475aca4de47c34ea7abde619e2a46b8b9f91f5d7846e75a89db72"
Oct 30 20:18:03.713659 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:03.713635    4823 scope.go:122] "RemoveContainer" containerID="f44290dcf547bb659391719ef4ef0cb33cfd89e4dc7b54ff8c85d5249b0bf4e7"
Oct 30 20:18:03.722022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:03.722000    4823 scope.go:122] "RemoveContainer" containerID="59c661798bab69bb7e291a2abcb48bd684b8155ea9f39ba0f7decdbee8fdc492"
Oct 30 20:18:03.730067 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:03.730046    4823 scope.go:122] "RemoveContainer" containerID="435434b095f86254450d0d239e3b672da8bcd0eeeb7aec4129f7b174015f6f29"
Oct 30 20:18:03.738518 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:03.738492    4823 scope.go:122] "RemoveContainer" containerID="a6c419e795fc117e9d507b750e7b8385b00991a96439ab3b30218b2609903e96"
Oct 30 20:18:03.746694 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:03.746676    4823 scope.go:122] "RemoveContainer" containerID="d014794c049f4b152d9c8aef77e87ddb60bdbbd8091e9db149b50638bcf8bf34"
Oct 30 20:18:03.754614 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:03.754589    4823 scope.go:122] "RemoveContainer" containerID="dcfd734d173dd9ef6516c70bbb935af53d62e1bc336e475907dc3a90cccf675f"
Oct 30 20:18:03.762998 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:03.762979    4823 scope.go:122] "RemoveContainer" containerID="e2abc4eaecb52061e711df5a72c884318de6567ceef0a38e09e55a5ef59d3056"
Oct 30 20:18:03.771115 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:03.771093    4823 scope.go:122] "RemoveContainer" containerID="f55e49f7b7532e7935ee3ca99aa94180bc95029f3963ac17f883e4b347c4ff24"
Oct 30 20:18:03.779288 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:03.779257    4823 scope.go:122] "RemoveContainer" containerID="de2406bac5ae22ebc9167f54edcc6ed76fb912c611ccacf5474b01e4fa18be6a"
Oct 30 20:18:03.788496 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:03.788479    4823 scope.go:122] "RemoveContainer" containerID="0f78691a19cc1fc287c341140b35ad70d1500105e3ebb2ff77aa90623f33c023"
Oct 30 20:18:11.902257 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:11.902200    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:18:25.020192 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:25.020157    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:18:27.649058 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:27.649010    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8krz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:18:27.649580 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:27.649097    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9" podUID="8574d565-f870-4374-97ca-7e5bee83f2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:18:32.014469 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:18:32.014431    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:18:33.045380 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.045347    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"]
Oct 30 20:18:33.045843 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.045541    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm" podUID="f8eb246a-ee5d-4b22-8e01-f280a76b2521" containerName="discovery" containerID="cri-o://45049a452b079902f910fdff9d69a24cc91d955c163c6314a88aaba34f730f5d" gracePeriod=30
Oct 30 20:18:33.314284 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.314257    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 20:18:33.374377 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.374337    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-csr-ca-configmap\") pod \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") "
Oct 30 20:18:33.374585 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.374428    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-kubeconfig\") pod \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") "
Oct 30 20:18:33.374585 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.374527    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-token\" (UniqueName: \"kubernetes.io/projected/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-token\") pod \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") "
Oct 30 20:18:33.374585 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.374573    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-csr-dns-cert\") pod \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") "
Oct 30 20:18:33.374786 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.374607    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8eb246a-ee5d-4b22-8e01-f280a76b2521-kube-api-access-dg2zk\" (UniqueName: \"kubernetes.io/projected/f8eb246a-ee5d-4b22-8e01-f280a76b2521-kube-api-access-dg2zk\") pod \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") "
Oct 30 20:18:33.374786 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.374731    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-cacerts\" (UniqueName: \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-cacerts\") pod \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") "
Oct 30 20:18:33.374786 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.374777    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eb246a-ee5d-4b22-8e01-f280a76b2521-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eb246a-ee5d-4b22-8e01-f280a76b2521-local-certs\") pod \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\" (UID: \"f8eb246a-ee5d-4b22-8e01-f280a76b2521\") "
Oct 30 20:18:33.375002 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.374846    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-csr-ca-configmap" pod "f8eb246a-ee5d-4b22-8e01-f280a76b2521" (UID: "f8eb246a-ee5d-4b22-8e01-f280a76b2521"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:33.375210 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.375171    4823 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-csr-ca-configmap\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:33.377025 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.376995    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-token" pod "f8eb246a-ee5d-4b22-8e01-f280a76b2521" (UID: "f8eb246a-ee5d-4b22-8e01-f280a76b2521"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:33.377343 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.377303    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-csr-dns-cert" pod "f8eb246a-ee5d-4b22-8e01-f280a76b2521" (UID: "f8eb246a-ee5d-4b22-8e01-f280a76b2521"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:33.377443 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.377373    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eb246a-ee5d-4b22-8e01-f280a76b2521-local-certs" pod "f8eb246a-ee5d-4b22-8e01-f280a76b2521" (UID: "f8eb246a-ee5d-4b22-8e01-f280a76b2521"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:33.377491 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.377469    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-kubeconfig" pod "f8eb246a-ee5d-4b22-8e01-f280a76b2521" (UID: "f8eb246a-ee5d-4b22-8e01-f280a76b2521"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:33.377592 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.377571    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-cacerts" pod "f8eb246a-ee5d-4b22-8e01-f280a76b2521" (UID: "f8eb246a-ee5d-4b22-8e01-f280a76b2521"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:33.377638 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.377616    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8eb246a-ee5d-4b22-8e01-f280a76b2521-kube-api-access-dg2zk" pod "f8eb246a-ee5d-4b22-8e01-f280a76b2521" (UID: "f8eb246a-ee5d-4b22-8e01-f280a76b2521"). InnerVolumeSpecName "kube-api-access-dg2zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:33.476509 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.476472    4823 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-cacerts\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:33.476509 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.476504    4823 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eb246a-ee5d-4b22-8e01-f280a76b2521-local-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:33.476509 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.476515    4823 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-kubeconfig\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:33.476751 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.476525    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:33.476751 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.476533    4823 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f8eb246a-ee5d-4b22-8e01-f280a76b2521-istio-csr-dns-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:33.476751 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:33.476543    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg2zk\" (UniqueName: \"kubernetes.io/projected/f8eb246a-ee5d-4b22-8e01-f280a76b2521-kube-api-access-dg2zk\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:34.060241 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:34.060209    4823 generic.go:356] "Generic (PLEG): container finished" podID="f8eb246a-ee5d-4b22-8e01-f280a76b2521" containerID="45049a452b079902f910fdff9d69a24cc91d955c163c6314a88aaba34f730f5d" exitCode=0
Oct 30 20:18:34.060783 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:34.060273    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"
Oct 30 20:18:34.060783 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:34.060289    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm" event={"ID":"f8eb246a-ee5d-4b22-8e01-f280a76b2521","Type":"ContainerDied","Data":"45049a452b079902f910fdff9d69a24cc91d955c163c6314a88aaba34f730f5d"}
Oct 30 20:18:34.060783 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:34.060368    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm" event={"ID":"f8eb246a-ee5d-4b22-8e01-f280a76b2521","Type":"ContainerDied","Data":"4ca914f960ca27daf63b9a1a11201eaa1ae6745bd12429faf2e126a2571bc96b"}
Oct 30 20:18:34.060783 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:34.060393    4823 scope.go:122] "RemoveContainer" containerID="45049a452b079902f910fdff9d69a24cc91d955c163c6314a88aaba34f730f5d"
Oct 30 20:18:34.073556 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:34.073534    4823 scope.go:122] "RemoveContainer" containerID="45049a452b079902f910fdff9d69a24cc91d955c163c6314a88aaba34f730f5d"
Oct 30 20:18:34.073829 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:18:34.073807    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45049a452b079902f910fdff9d69a24cc91d955c163c6314a88aaba34f730f5d\": container with ID starting with 45049a452b079902f910fdff9d69a24cc91d955c163c6314a88aaba34f730f5d not found: ID does not exist" containerID="45049a452b079902f910fdff9d69a24cc91d955c163c6314a88aaba34f730f5d"
Oct 30 20:18:34.073950 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:34.073841    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45049a452b079902f910fdff9d69a24cc91d955c163c6314a88aaba34f730f5d"} err="failed to get container status \"45049a452b079902f910fdff9d69a24cc91d955c163c6314a88aaba34f730f5d\": rpc error: code = NotFound desc = could not find container \"45049a452b079902f910fdff9d69a24cc91d955c163c6314a88aaba34f730f5d\": container with ID starting with 45049a452b079902f910fdff9d69a24cc91d955c163c6314a88aaba34f730f5d not found: ID does not exist"
Oct 30 20:18:34.089608 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:34.089573    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"]
Oct 30 20:18:34.092723 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:34.092700    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l8kcm"]
Oct 30 20:18:34.277706 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:34.277674    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8eb246a-ee5d-4b22-8e01-f280a76b2521" path="/var/lib/kubelet/pods/f8eb246a-ee5d-4b22-8e01-f280a76b2521/volumes"
Oct 30 20:18:38.552208 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.552176    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"]
Oct 30 20:18:38.552675 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.552661    4823 cpu_manager.go:414] "containerMap: removing container" podUID="f8eb246a-ee5d-4b22-8e01-f280a76b2521" containerName="discovery"
Oct 30 20:18:38.552724 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.552676    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8eb246a-ee5d-4b22-8e01-f280a76b2521" containerName="discovery"
Oct 30 20:18:38.552789 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.552780    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8eb246a-ee5d-4b22-8e01-f280a76b2521" containerName="discovery"
Oct 30 20:18:38.557511 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.557491    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"
Oct 30 20:18:38.563763 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.563737    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:18:38.563928 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.563758    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:18:38.567130 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.567107    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"]
Oct 30 20:18:38.609098 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.609073    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:18:38.609098 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.609099    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:18:38.624391 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.624368    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5g8vl\" (UID: \"c1c2f926-11c7-43d5-bd4b-e0319700e5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"
Oct 30 20:18:38.624532 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.624507    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5g8vl\" (UID: \"c1c2f926-11c7-43d5-bd4b-e0319700e5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"
Oct 30 20:18:38.624578 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.624540    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh9kh\" (UniqueName: \"kubernetes.io/projected/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-kube-api-access-qh9kh\") pod \"maas-api-revoke-sub-046e1343-5g8vl\" (UID: \"c1c2f926-11c7-43d5-bd4b-e0319700e5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"
Oct 30 20:18:38.725071 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.725041    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5g8vl\" (UID: \"c1c2f926-11c7-43d5-bd4b-e0319700e5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"
Oct 30 20:18:38.725071 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.725078    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh9kh\" (UniqueName: \"kubernetes.io/projected/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-kube-api-access-qh9kh\") pod \"maas-api-revoke-sub-046e1343-5g8vl\" (UID: \"c1c2f926-11c7-43d5-bd4b-e0319700e5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"
Oct 30 20:18:38.725264 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.725121    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5g8vl\" (UID: \"c1c2f926-11c7-43d5-bd4b-e0319700e5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"
Oct 30 20:18:38.725693 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.725673    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5g8vl\" (UID: \"c1c2f926-11c7-43d5-bd4b-e0319700e5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"
Oct 30 20:18:38.727484 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.727461    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5g8vl\" (UID: \"c1c2f926-11c7-43d5-bd4b-e0319700e5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"
Oct 30 20:18:38.736895 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.736851    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh9kh\" (UniqueName: \"kubernetes.io/projected/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-kube-api-access-qh9kh\") pod \"maas-api-revoke-sub-046e1343-5g8vl\" (UID: \"c1c2f926-11c7-43d5-bd4b-e0319700e5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"
Oct 30 20:18:38.872158 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:38.872134    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"
Oct 30 20:18:39.158442 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:39.158378    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"]
Oct 30 20:18:39.161183 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:18:39.161155    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c2f926_11c7_43d5_bd4b_e0319700e5f6.slice/crio-9ff0a2e123bb5ba1f9b85c5c41801e66c9fab2dc9f5ec2a280b1c45fc9bec19a WatchSource:0}: Error finding container 9ff0a2e123bb5ba1f9b85c5c41801e66c9fab2dc9f5ec2a280b1c45fc9bec19a: Status 404 returned error can't find the container with id 9ff0a2e123bb5ba1f9b85c5c41801e66c9fab2dc9f5ec2a280b1c45fc9bec19a
Oct 30 20:18:40.106619 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:40.106584    4823 generic.go:356] "Generic (PLEG): container finished" podID="c1c2f926-11c7-43d5-bd4b-e0319700e5f6" containerID="824cc3b5517b16d7eaf42e38fba23c468564ff4526d7175816d660b14f45aed6" exitCode=0
Oct 30 20:18:40.107107 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:40.106670    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl" event={"ID":"c1c2f926-11c7-43d5-bd4b-e0319700e5f6","Type":"ContainerDied","Data":"824cc3b5517b16d7eaf42e38fba23c468564ff4526d7175816d660b14f45aed6"}
Oct 30 20:18:40.107107 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:40.106741    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl" event={"ID":"c1c2f926-11c7-43d5-bd4b-e0319700e5f6","Type":"ContainerStarted","Data":"9ff0a2e123bb5ba1f9b85c5c41801e66c9fab2dc9f5ec2a280b1c45fc9bec19a"}
Oct 30 20:18:41.255800 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:41.255776    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"
Oct 30 20:18:41.350123 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:41.350093    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-kube-api-access-qh9kh\" (UniqueName: \"kubernetes.io/projected/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-kube-api-access-qh9kh\") pod \"c1c2f926-11c7-43d5-bd4b-e0319700e5f6\" (UID: \"c1c2f926-11c7-43d5-bd4b-e0319700e5f6\") "
Oct 30 20:18:41.350309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:41.350130    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-maas-api-openshift-ca\") pod \"c1c2f926-11c7-43d5-bd4b-e0319700e5f6\" (UID: \"c1c2f926-11c7-43d5-bd4b-e0319700e5f6\") "
Oct 30 20:18:41.350309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:41.350194    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-maas-api-xks-ca\") pod \"c1c2f926-11c7-43d5-bd4b-e0319700e5f6\" (UID: \"c1c2f926-11c7-43d5-bd4b-e0319700e5f6\") "
Oct 30 20:18:41.350552 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:41.350529    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-maas-api-openshift-ca" pod "c1c2f926-11c7-43d5-bd4b-e0319700e5f6" (UID: "c1c2f926-11c7-43d5-bd4b-e0319700e5f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:41.352379 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:41.352346    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-kube-api-access-qh9kh" pod "c1c2f926-11c7-43d5-bd4b-e0319700e5f6" (UID: "c1c2f926-11c7-43d5-bd4b-e0319700e5f6"). InnerVolumeSpecName "kube-api-access-qh9kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:41.352379 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:41.352364    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-maas-api-xks-ca" pod "c1c2f926-11c7-43d5-bd4b-e0319700e5f6" (UID: "c1c2f926-11c7-43d5-bd4b-e0319700e5f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:41.451019 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:41.450936    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:41.451019 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:41.450969    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh9kh\" (UniqueName: \"kubernetes.io/projected/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-kube-api-access-qh9kh\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:41.451019 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:41.450983    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2f926-11c7-43d5-bd4b-e0319700e5f6-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:42.120081 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:42.120046    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"
Oct 30 20:18:42.120263 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:42.120080    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl" event={"ID":"c1c2f926-11c7-43d5-bd4b-e0319700e5f6","Type":"ContainerDied","Data":"9ff0a2e123bb5ba1f9b85c5c41801e66c9fab2dc9f5ec2a280b1c45fc9bec19a"}
Oct 30 20:18:42.120263 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:42.120131    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ff0a2e123bb5ba1f9b85c5c41801e66c9fab2dc9f5ec2a280b1c45fc9bec19a"
Oct 30 20:18:48.881894 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:48.881833    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"]
Oct 30 20:18:48.882649 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:48.882625    4823 cpu_manager.go:414] "containerMap: removing container" podUID="c1c2f926-11c7-43d5-bd4b-e0319700e5f6" containerName="revoke-keys"
Oct 30 20:18:48.882710 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:48.882654    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c2f926-11c7-43d5-bd4b-e0319700e5f6" containerName="revoke-keys"
Oct 30 20:18:48.882890 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:48.882856    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c2f926-11c7-43d5-bd4b-e0319700e5f6" containerName="revoke-keys"
Oct 30 20:18:48.886920 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:48.886895    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"
Oct 30 20:18:48.896327 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:48.895790    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"]
Oct 30 20:18:48.896919 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:48.896894    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:18:48.897488 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:48.897471    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:18:48.897621 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:48.897569    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:18:48.897699 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:48.897655    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:18:48.923794 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:48.923763    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9967e77-6709-4ea7-8908-fd430dac7e6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-zhvjg\" (UID: \"d9967e77-6709-4ea7-8908-fd430dac7e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"
Oct 30 20:18:48.924001 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:48.923815    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlr7t\" (UniqueName: \"kubernetes.io/projected/d9967e77-6709-4ea7-8908-fd430dac7e6f-kube-api-access-hlr7t\") pod \"maas-api-revoke-sub-d3fba056-zhvjg\" (UID: \"d9967e77-6709-4ea7-8908-fd430dac7e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"
Oct 30 20:18:48.924001 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:48.923923    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9967e77-6709-4ea7-8908-fd430dac7e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-zhvjg\" (UID: \"d9967e77-6709-4ea7-8908-fd430dac7e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"
Oct 30 20:18:49.025216 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:49.025184    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9967e77-6709-4ea7-8908-fd430dac7e6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-zhvjg\" (UID: \"d9967e77-6709-4ea7-8908-fd430dac7e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"
Oct 30 20:18:49.025381 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:49.025227    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlr7t\" (UniqueName: \"kubernetes.io/projected/d9967e77-6709-4ea7-8908-fd430dac7e6f-kube-api-access-hlr7t\") pod \"maas-api-revoke-sub-d3fba056-zhvjg\" (UID: \"d9967e77-6709-4ea7-8908-fd430dac7e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"
Oct 30 20:18:49.025381 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:49.025257    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9967e77-6709-4ea7-8908-fd430dac7e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-zhvjg\" (UID: \"d9967e77-6709-4ea7-8908-fd430dac7e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"
Oct 30 20:18:49.026059 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:49.026026    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9967e77-6709-4ea7-8908-fd430dac7e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-zhvjg\" (UID: \"d9967e77-6709-4ea7-8908-fd430dac7e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"
Oct 30 20:18:49.027547 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:49.027523    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9967e77-6709-4ea7-8908-fd430dac7e6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-zhvjg\" (UID: \"d9967e77-6709-4ea7-8908-fd430dac7e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"
Oct 30 20:18:49.039354 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:49.039324    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlr7t\" (UniqueName: \"kubernetes.io/projected/d9967e77-6709-4ea7-8908-fd430dac7e6f-kube-api-access-hlr7t\") pod \"maas-api-revoke-sub-d3fba056-zhvjg\" (UID: \"d9967e77-6709-4ea7-8908-fd430dac7e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"
Oct 30 20:18:49.204060 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:49.203976    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"
Oct 30 20:18:49.491496 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:49.491425    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"]
Oct 30 20:18:49.494943 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:18:49.494915    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9967e77_6709_4ea7_8908_fd430dac7e6f.slice/crio-69b39699838a13314e46a77e48f5d3e6b7f253ad4f40177ad98bce3ac2473737 WatchSource:0}: Error finding container 69b39699838a13314e46a77e48f5d3e6b7f253ad4f40177ad98bce3ac2473737: Status 404 returned error can't find the container with id 69b39699838a13314e46a77e48f5d3e6b7f253ad4f40177ad98bce3ac2473737
Oct 30 20:18:49.694503 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:49.694471    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:18:50.172846 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:50.172805    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg" event={"ID":"d9967e77-6709-4ea7-8908-fd430dac7e6f","Type":"ContainerStarted","Data":"1b4c1a93a51bba28fc3eaf66e42170cc5a365a6f917f588c1e40f3368ccf10e7"}
Oct 30 20:18:50.173407 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:50.172885    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg" event={"ID":"d9967e77-6709-4ea7-8908-fd430dac7e6f","Type":"ContainerStarted","Data":"69b39699838a13314e46a77e48f5d3e6b7f253ad4f40177ad98bce3ac2473737"}
Oct 30 20:18:50.192562 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:50.192505    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg" podStartSLOduration=2.192492149 podStartE2EDuration="2.192492149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:50.189899018 +0000 UTC m=+2930.082513345" watchObservedRunningTime="2026-10-30 20:18:50.192492149 +0000 UTC m=+2930.085106465"
Oct 30 20:18:51.219887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:51.218833    4823 generic.go:356] "Generic (PLEG): container finished" podID="d9967e77-6709-4ea7-8908-fd430dac7e6f" containerID="1b4c1a93a51bba28fc3eaf66e42170cc5a365a6f917f588c1e40f3368ccf10e7" exitCode=0
Oct 30 20:18:51.219887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:51.219026    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg" event={"ID":"d9967e77-6709-4ea7-8908-fd430dac7e6f","Type":"ContainerDied","Data":"1b4c1a93a51bba28fc3eaf66e42170cc5a365a6f917f588c1e40f3368ccf10e7"}
Oct 30 20:18:52.558228 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:52.558107    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"
Oct 30 20:18:52.645942 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:52.645909    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9967e77-6709-4ea7-8908-fd430dac7e6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9967e77-6709-4ea7-8908-fd430dac7e6f-maas-api-xks-ca\") pod \"d9967e77-6709-4ea7-8908-fd430dac7e6f\" (UID: \"d9967e77-6709-4ea7-8908-fd430dac7e6f\") "
Oct 30 20:18:52.646142 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:52.646004    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9967e77-6709-4ea7-8908-fd430dac7e6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9967e77-6709-4ea7-8908-fd430dac7e6f-maas-api-openshift-ca\") pod \"d9967e77-6709-4ea7-8908-fd430dac7e6f\" (UID: \"d9967e77-6709-4ea7-8908-fd430dac7e6f\") "
Oct 30 20:18:52.646142 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:52.646039    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9967e77-6709-4ea7-8908-fd430dac7e6f-kube-api-access-hlr7t\" (UniqueName: \"kubernetes.io/projected/d9967e77-6709-4ea7-8908-fd430dac7e6f-kube-api-access-hlr7t\") pod \"d9967e77-6709-4ea7-8908-fd430dac7e6f\" (UID: \"d9967e77-6709-4ea7-8908-fd430dac7e6f\") "
Oct 30 20:18:52.646421 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:52.646396    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9967e77-6709-4ea7-8908-fd430dac7e6f-maas-api-openshift-ca" pod "d9967e77-6709-4ea7-8908-fd430dac7e6f" (UID: "d9967e77-6709-4ea7-8908-fd430dac7e6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:52.648206 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:52.648180    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9967e77-6709-4ea7-8908-fd430dac7e6f-kube-api-access-hlr7t" pod "d9967e77-6709-4ea7-8908-fd430dac7e6f" (UID: "d9967e77-6709-4ea7-8908-fd430dac7e6f"). InnerVolumeSpecName "kube-api-access-hlr7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:52.648316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:52.648206    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9967e77-6709-4ea7-8908-fd430dac7e6f-maas-api-xks-ca" pod "d9967e77-6709-4ea7-8908-fd430dac7e6f" (UID: "d9967e77-6709-4ea7-8908-fd430dac7e6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:18:52.746916 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:52.746857    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9967e77-6709-4ea7-8908-fd430dac7e6f-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:52.746916 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:52.746914    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlr7t\" (UniqueName: \"kubernetes.io/projected/d9967e77-6709-4ea7-8908-fd430dac7e6f-kube-api-access-hlr7t\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:52.747117 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:52.746927    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9967e77-6709-4ea7-8908-fd430dac7e6f-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:53.238941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:53.238901    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg" event={"ID":"d9967e77-6709-4ea7-8908-fd430dac7e6f","Type":"ContainerDied","Data":"69b39699838a13314e46a77e48f5d3e6b7f253ad4f40177ad98bce3ac2473737"}
Oct 30 20:18:53.239097 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:53.238956    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69b39699838a13314e46a77e48f5d3e6b7f253ad4f40177ad98bce3ac2473737"
Oct 30 20:18:53.239097 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:53.238957    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"
Oct 30 20:18:56.026842 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:56.026806    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"]
Oct 30 20:18:56.034926 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:56.034889    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f75163a5-8hx99"]
Oct 30 20:18:56.276987 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:56.276908    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd839149-8249-44f4-baf7-9fcedbb58670" path="/var/lib/kubelet/pods/fd839149-8249-44f4-baf7-9fcedbb58670/volumes"
Oct 30 20:18:59.812435 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:18:59.812397    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:19:02.013366 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:19:02.013325    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:19:03.879887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:03.879213    4823 scope.go:122] "RemoveContainer" containerID="21bf52077725f14644124cef293c84926529b0494e85fa858632ddd68cff0de9"
Oct 30 20:19:03.935992 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:03.935947    4823 scope.go:122] "RemoveContainer" containerID="14c154fc891c20f1a5ddfeeae86e3e960f9644ae84bf17e423deff548d54bcd8"
Oct 30 20:19:03.945395 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:03.945371    4823 scope.go:122] "RemoveContainer" containerID="ad934b8ce228338c98bc334cec9dc4f4eb0505b59e760936d997fe6039191a22"
Oct 30 20:19:03.982884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:03.979093    4823 scope.go:122] "RemoveContainer" containerID="60c94502bd743d2c3f091b0e7ccb7678c4ecbf490d8c0ea627a5f295bb334bbe"
Oct 30 20:19:04.028055 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:04.028022    4823 scope.go:122] "RemoveContainer" containerID="e1bc0a989637882fe0f9795fe20eca35d4268bcc7602b0c70b092d8d5165cdd6"
Oct 30 20:19:04.038928 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:04.038909    4823 scope.go:122] "RemoveContainer" containerID="b56825a4a29616189d82e606cc8a7d5dcd41373669781332b7d4bea9f6a927d5"
Oct 30 20:19:04.047067 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:04.047045    4823 scope.go:122] "RemoveContainer" containerID="dabef7919df01b48ba17e2eb6991a22a90017fcce8fdcd28f0f63fa7eb6b8bda"
Oct 30 20:19:04.089172 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:04.089149    4823 scope.go:122] "RemoveContainer" containerID="6a01f7d35bcccddde6bae38d00982232b557af285240438d43bc462736009a58"
Oct 30 20:19:04.127843 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:04.127489    4823 scope.go:122] "RemoveContainer" containerID="f5ed87df1903902e1f958108bcfecf7e40bbd502d156a2d0b51d4b80b461cb50"
Oct 30 20:19:04.139810 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:04.139788    4823 scope.go:122] "RemoveContainer" containerID="da6449a24ab460eab84af1f4d3bf9aa737a7b7e80b4f225beb7aa007ac49d4f2"
Oct 30 20:19:04.148253 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:04.148229    4823 scope.go:122] "RemoveContainer" containerID="148afa2990ea874183769d17a1a27b40b5e40730b77f0929847ea2e6772efdd1"
Oct 30 20:19:04.329890 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:04.329205    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:19:09.782503 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.782464    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"]
Oct 30 20:19:09.782950 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.782936    4823 cpu_manager.go:414] "containerMap: removing container" podUID="d9967e77-6709-4ea7-8908-fd430dac7e6f" containerName="revoke-keys"
Oct 30 20:19:09.783003 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.782951    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9967e77-6709-4ea7-8908-fd430dac7e6f" containerName="revoke-keys"
Oct 30 20:19:09.783056 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.783046    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9967e77-6709-4ea7-8908-fd430dac7e6f" containerName="revoke-keys"
Oct 30 20:19:09.788403 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.788379    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"
Oct 30 20:19:09.793329 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.793306    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:19:09.793474 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.793379    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:09.796445 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.796419    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"]
Oct 30 20:19:09.800158 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.800137    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:19:09.800300 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.800284    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:19:09.878587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.878545    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a22111-bc74-4b1b-962f-8940f3dedd1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mc78q\" (UID: \"d0a22111-bc74-4b1b-962f-8940f3dedd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"
Oct 30 20:19:09.878814 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.878618    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a22111-bc74-4b1b-962f-8940f3dedd1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mc78q\" (UID: \"d0a22111-bc74-4b1b-962f-8940f3dedd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"
Oct 30 20:19:09.878814 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.878662    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmbdl\" (UniqueName: \"kubernetes.io/projected/d0a22111-bc74-4b1b-962f-8940f3dedd1e-kube-api-access-dmbdl\") pod \"maas-api-revoke-sub-ae594979-mc78q\" (UID: \"d0a22111-bc74-4b1b-962f-8940f3dedd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"
Oct 30 20:19:09.979696 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.979656    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a22111-bc74-4b1b-962f-8940f3dedd1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mc78q\" (UID: \"d0a22111-bc74-4b1b-962f-8940f3dedd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"
Oct 30 20:19:09.979696 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.979698    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmbdl\" (UniqueName: \"kubernetes.io/projected/d0a22111-bc74-4b1b-962f-8940f3dedd1e-kube-api-access-dmbdl\") pod \"maas-api-revoke-sub-ae594979-mc78q\" (UID: \"d0a22111-bc74-4b1b-962f-8940f3dedd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"
Oct 30 20:19:09.980002 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.979787    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a22111-bc74-4b1b-962f-8940f3dedd1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mc78q\" (UID: \"d0a22111-bc74-4b1b-962f-8940f3dedd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"
Oct 30 20:19:09.980481 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.980451    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a22111-bc74-4b1b-962f-8940f3dedd1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mc78q\" (UID: \"d0a22111-bc74-4b1b-962f-8940f3dedd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"
Oct 30 20:19:09.982192 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.982169    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a22111-bc74-4b1b-962f-8940f3dedd1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mc78q\" (UID: \"d0a22111-bc74-4b1b-962f-8940f3dedd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"
Oct 30 20:19:09.992230 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:09.992203    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmbdl\" (UniqueName: \"kubernetes.io/projected/d0a22111-bc74-4b1b-962f-8940f3dedd1e-kube-api-access-dmbdl\") pod \"maas-api-revoke-sub-ae594979-mc78q\" (UID: \"d0a22111-bc74-4b1b-962f-8940f3dedd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"
Oct 30 20:19:10.103513 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:10.103434    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"
Oct 30 20:19:10.382804 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:10.382774    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"]
Oct 30 20:19:10.386177 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:19:10.386144    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a22111_bc74_4b1b_962f_8940f3dedd1e.slice/crio-bc3fb25ebda2dc6dd97c50307f5d31b8628a008b531a4aa45bd22df7a13ad7c1 WatchSource:0}: Error finding container bc3fb25ebda2dc6dd97c50307f5d31b8628a008b531a4aa45bd22df7a13ad7c1: Status 404 returned error can't find the container with id bc3fb25ebda2dc6dd97c50307f5d31b8628a008b531a4aa45bd22df7a13ad7c1
Oct 30 20:19:11.377568 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:11.377532    4823 generic.go:356] "Generic (PLEG): container finished" podID="d0a22111-bc74-4b1b-962f-8940f3dedd1e" containerID="497bd4dec29bcab95d71347f6bb57a372e9dcc7880f9fba8e3aa0667aaa0e6ed" exitCode=0
Oct 30 20:19:11.377980 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:11.377588    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q" event={"ID":"d0a22111-bc74-4b1b-962f-8940f3dedd1e","Type":"ContainerDied","Data":"497bd4dec29bcab95d71347f6bb57a372e9dcc7880f9fba8e3aa0667aaa0e6ed"}
Oct 30 20:19:11.377980 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:11.377642    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q" event={"ID":"d0a22111-bc74-4b1b-962f-8940f3dedd1e","Type":"ContainerStarted","Data":"bc3fb25ebda2dc6dd97c50307f5d31b8628a008b531a4aa45bd22df7a13ad7c1"}
Oct 30 20:19:12.529411 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:12.529386    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"
Oct 30 20:19:12.605857 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:12.605828    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a22111-bc74-4b1b-962f-8940f3dedd1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a22111-bc74-4b1b-962f-8940f3dedd1e-maas-api-openshift-ca\") pod \"d0a22111-bc74-4b1b-962f-8940f3dedd1e\" (UID: \"d0a22111-bc74-4b1b-962f-8940f3dedd1e\") "
Oct 30 20:19:12.606049 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:12.605950    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a22111-bc74-4b1b-962f-8940f3dedd1e-kube-api-access-dmbdl\" (UniqueName: \"kubernetes.io/projected/d0a22111-bc74-4b1b-962f-8940f3dedd1e-kube-api-access-dmbdl\") pod \"d0a22111-bc74-4b1b-962f-8940f3dedd1e\" (UID: \"d0a22111-bc74-4b1b-962f-8940f3dedd1e\") "
Oct 30 20:19:12.606049 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:12.605997    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0a22111-bc74-4b1b-962f-8940f3dedd1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a22111-bc74-4b1b-962f-8940f3dedd1e-maas-api-xks-ca\") pod \"d0a22111-bc74-4b1b-962f-8940f3dedd1e\" (UID: \"d0a22111-bc74-4b1b-962f-8940f3dedd1e\") "
Oct 30 20:19:12.606256 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:12.606226    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a22111-bc74-4b1b-962f-8940f3dedd1e-maas-api-openshift-ca" pod "d0a22111-bc74-4b1b-962f-8940f3dedd1e" (UID: "d0a22111-bc74-4b1b-962f-8940f3dedd1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:12.608114 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:12.608086    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a22111-bc74-4b1b-962f-8940f3dedd1e-maas-api-xks-ca" pod "d0a22111-bc74-4b1b-962f-8940f3dedd1e" (UID: "d0a22111-bc74-4b1b-962f-8940f3dedd1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:12.608114 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:12.608108    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a22111-bc74-4b1b-962f-8940f3dedd1e-kube-api-access-dmbdl" pod "d0a22111-bc74-4b1b-962f-8940f3dedd1e" (UID: "d0a22111-bc74-4b1b-962f-8940f3dedd1e"). InnerVolumeSpecName "kube-api-access-dmbdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:12.707666 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:12.707599    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmbdl\" (UniqueName: \"kubernetes.io/projected/d0a22111-bc74-4b1b-962f-8940f3dedd1e-kube-api-access-dmbdl\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:12.707666 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:12.707623    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a22111-bc74-4b1b-962f-8940f3dedd1e-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:12.707666 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:12.707635    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a22111-bc74-4b1b-962f-8940f3dedd1e-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:13.390456 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:13.390418    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q" event={"ID":"d0a22111-bc74-4b1b-962f-8940f3dedd1e","Type":"ContainerDied","Data":"bc3fb25ebda2dc6dd97c50307f5d31b8628a008b531a4aa45bd22df7a13ad7c1"}
Oct 30 20:19:13.390456 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:13.390446    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"
Oct 30 20:19:13.390673 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:13.390474    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc3fb25ebda2dc6dd97c50307f5d31b8628a008b531a4aa45bd22df7a13ad7c1"
Oct 30 20:19:20.689433 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:20.689393    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:19:24.504886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:24.504270    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:19:28.138973 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.138939    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"]
Oct 30 20:19:28.139466 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.139449    4823 cpu_manager.go:414] "containerMap: removing container" podUID="d0a22111-bc74-4b1b-962f-8940f3dedd1e" containerName="revoke-keys"
Oct 30 20:19:28.139524 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.139467    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a22111-bc74-4b1b-962f-8940f3dedd1e" containerName="revoke-keys"
Oct 30 20:19:28.139579 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.139554    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a22111-bc74-4b1b-962f-8940f3dedd1e" containerName="revoke-keys"
Oct 30 20:19:28.143892 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.143851    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"
Oct 30 20:19:28.154666 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.154637    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:19:28.154820 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.154731    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:19:28.154820 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.154762    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:28.154982 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.154857    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:19:28.156066 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.156037    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"]
Oct 30 20:19:28.168104 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.167327    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/340190c8-9e15-4c76-b559-c84d10ef02d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2sx5c\" (UID: \"340190c8-9e15-4c76-b559-c84d10ef02d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"
Oct 30 20:19:28.168664 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.168636    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2mds\" (UniqueName: \"kubernetes.io/projected/340190c8-9e15-4c76-b559-c84d10ef02d1-kube-api-access-m2mds\") pod \"maas-api-revoke-sub-a4aef7d7-2sx5c\" (UID: \"340190c8-9e15-4c76-b559-c84d10ef02d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"
Oct 30 20:19:28.168800 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.168686    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/340190c8-9e15-4c76-b559-c84d10ef02d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2sx5c\" (UID: \"340190c8-9e15-4c76-b559-c84d10ef02d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"
Oct 30 20:19:28.269224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.269185    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/340190c8-9e15-4c76-b559-c84d10ef02d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2sx5c\" (UID: \"340190c8-9e15-4c76-b559-c84d10ef02d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"
Oct 30 20:19:28.269418 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.269256    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/340190c8-9e15-4c76-b559-c84d10ef02d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2sx5c\" (UID: \"340190c8-9e15-4c76-b559-c84d10ef02d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"
Oct 30 20:19:28.269418 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.269314    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2mds\" (UniqueName: \"kubernetes.io/projected/340190c8-9e15-4c76-b559-c84d10ef02d1-kube-api-access-m2mds\") pod \"maas-api-revoke-sub-a4aef7d7-2sx5c\" (UID: \"340190c8-9e15-4c76-b559-c84d10ef02d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"
Oct 30 20:19:28.269965 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.269939    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/340190c8-9e15-4c76-b559-c84d10ef02d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2sx5c\" (UID: \"340190c8-9e15-4c76-b559-c84d10ef02d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"
Oct 30 20:19:28.271782 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.271754    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/340190c8-9e15-4c76-b559-c84d10ef02d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2sx5c\" (UID: \"340190c8-9e15-4c76-b559-c84d10ef02d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"
Oct 30 20:19:28.282070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.282048    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2mds\" (UniqueName: \"kubernetes.io/projected/340190c8-9e15-4c76-b559-c84d10ef02d1-kube-api-access-m2mds\") pod \"maas-api-revoke-sub-a4aef7d7-2sx5c\" (UID: \"340190c8-9e15-4c76-b559-c84d10ef02d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"
Oct 30 20:19:28.458994 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.458913    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"
Oct 30 20:19:28.738690 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:28.738663    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"]
Oct 30 20:19:28.741007 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:19:28.740981    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340190c8_9e15_4c76_b559_c84d10ef02d1.slice/crio-b6f2f68e2ad96c3e6e654183d49a6d85d59251256b512d15bd9e4a5e7a9159a1 WatchSource:0}: Error finding container b6f2f68e2ad96c3e6e654183d49a6d85d59251256b512d15bd9e4a5e7a9159a1: Status 404 returned error can't find the container with id b6f2f68e2ad96c3e6e654183d49a6d85d59251256b512d15bd9e4a5e7a9159a1
Oct 30 20:19:29.479078 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:29.479041    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c" event={"ID":"340190c8-9e15-4c76-b559-c84d10ef02d1","Type":"ContainerStarted","Data":"381c86f5de906154c18bc83773e922c6e79131f9412a865e36e6e63ab6f19592"}
Oct 30 20:19:29.479581 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:29.479101    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c" event={"ID":"340190c8-9e15-4c76-b559-c84d10ef02d1","Type":"ContainerStarted","Data":"b6f2f68e2ad96c3e6e654183d49a6d85d59251256b512d15bd9e4a5e7a9159a1"}
Oct 30 20:19:29.499571 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:29.499522    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c" podStartSLOduration=1.499508762 podStartE2EDuration="1.499508762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:29.497727935 +0000 UTC m=+2969.390342269" watchObservedRunningTime="2026-10-30 20:19:29.499508762 +0000 UTC m=+2969.392123086"
Oct 30 20:19:30.486099 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:30.486056    4823 generic.go:356] "Generic (PLEG): container finished" podID="340190c8-9e15-4c76-b559-c84d10ef02d1" containerID="381c86f5de906154c18bc83773e922c6e79131f9412a865e36e6e63ab6f19592" exitCode=0
Oct 30 20:19:30.486508 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:30.486152    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c" event={"ID":"340190c8-9e15-4c76-b559-c84d10ef02d1","Type":"ContainerDied","Data":"381c86f5de906154c18bc83773e922c6e79131f9412a865e36e6e63ab6f19592"}
Oct 30 20:19:31.631594 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:31.631569    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"
Oct 30 20:19:31.700210 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:31.700180    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/340190c8-9e15-4c76-b559-c84d10ef02d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/340190c8-9e15-4c76-b559-c84d10ef02d1-maas-api-xks-ca\") pod \"340190c8-9e15-4c76-b559-c84d10ef02d1\" (UID: \"340190c8-9e15-4c76-b559-c84d10ef02d1\") "
Oct 30 20:19:31.700385 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:31.700240    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/340190c8-9e15-4c76-b559-c84d10ef02d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/340190c8-9e15-4c76-b559-c84d10ef02d1-maas-api-openshift-ca\") pod \"340190c8-9e15-4c76-b559-c84d10ef02d1\" (UID: \"340190c8-9e15-4c76-b559-c84d10ef02d1\") "
Oct 30 20:19:31.700385 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:31.700363    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/340190c8-9e15-4c76-b559-c84d10ef02d1-kube-api-access-m2mds\" (UniqueName: \"kubernetes.io/projected/340190c8-9e15-4c76-b559-c84d10ef02d1-kube-api-access-m2mds\") pod \"340190c8-9e15-4c76-b559-c84d10ef02d1\" (UID: \"340190c8-9e15-4c76-b559-c84d10ef02d1\") "
Oct 30 20:19:31.700681 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:31.700655    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/340190c8-9e15-4c76-b559-c84d10ef02d1-maas-api-openshift-ca" pod "340190c8-9e15-4c76-b559-c84d10ef02d1" (UID: "340190c8-9e15-4c76-b559-c84d10ef02d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:31.702718 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:31.702695    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/340190c8-9e15-4c76-b559-c84d10ef02d1-maas-api-xks-ca" pod "340190c8-9e15-4c76-b559-c84d10ef02d1" (UID: "340190c8-9e15-4c76-b559-c84d10ef02d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:31.702821 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:31.702716    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340190c8-9e15-4c76-b559-c84d10ef02d1-kube-api-access-m2mds" pod "340190c8-9e15-4c76-b559-c84d10ef02d1" (UID: "340190c8-9e15-4c76-b559-c84d10ef02d1"). InnerVolumeSpecName "kube-api-access-m2mds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:31.801307 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:31.801228    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2mds\" (UniqueName: \"kubernetes.io/projected/340190c8-9e15-4c76-b559-c84d10ef02d1-kube-api-access-m2mds\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:31.801307 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:31.801256    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/340190c8-9e15-4c76-b559-c84d10ef02d1-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:31.801307 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:31.801266    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/340190c8-9e15-4c76-b559-c84d10ef02d1-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:32.014313 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:19:32.014267    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:19:32.500082 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:32.500045    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"
Oct 30 20:19:32.500265 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:32.500079    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c" event={"ID":"340190c8-9e15-4c76-b559-c84d10ef02d1","Type":"ContainerDied","Data":"b6f2f68e2ad96c3e6e654183d49a6d85d59251256b512d15bd9e4a5e7a9159a1"}
Oct 30 20:19:32.500265 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:32.500130    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6f2f68e2ad96c3e6e654183d49a6d85d59251256b512d15bd9e4a5e7a9159a1"
Oct 30 20:19:39.104895 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:39.104847    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:19:44.362127 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:44.362087    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:19:47.226662 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.226629    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"]
Oct 30 20:19:47.227113 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.227098    4823 cpu_manager.go:414] "containerMap: removing container" podUID="340190c8-9e15-4c76-b559-c84d10ef02d1" containerName="revoke-keys"
Oct 30 20:19:47.227166 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.227115    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340190c8-9e15-4c76-b559-c84d10ef02d1" containerName="revoke-keys"
Oct 30 20:19:47.227222 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.227213    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="340190c8-9e15-4c76-b559-c84d10ef02d1" containerName="revoke-keys"
Oct 30 20:19:47.231590 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.231573    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"
Oct 30 20:19:47.237822 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.237781    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:19:47.238309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.237823    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:47.238309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.237916    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:19:47.238309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.237996    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:19:47.241019 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.240994    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"]
Oct 30 20:19:47.268100 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.268065    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-qr6wl\" (UID: \"ceb0f5c7-f4ef-4e03-a500-64923d854fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"
Oct 30 20:19:47.268263 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.268184    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-qr6wl\" (UID: \"ceb0f5c7-f4ef-4e03-a500-64923d854fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"
Oct 30 20:19:47.268336 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.268313    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxjhb\" (UniqueName: \"kubernetes.io/projected/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-kube-api-access-mxjhb\") pod \"maas-api-revoke-sub-79805ca9-qr6wl\" (UID: \"ceb0f5c7-f4ef-4e03-a500-64923d854fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"
Oct 30 20:19:47.369256 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.369217    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxjhb\" (UniqueName: \"kubernetes.io/projected/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-kube-api-access-mxjhb\") pod \"maas-api-revoke-sub-79805ca9-qr6wl\" (UID: \"ceb0f5c7-f4ef-4e03-a500-64923d854fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"
Oct 30 20:19:47.369450 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.369299    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-qr6wl\" (UID: \"ceb0f5c7-f4ef-4e03-a500-64923d854fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"
Oct 30 20:19:47.369450 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.369350    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-qr6wl\" (UID: \"ceb0f5c7-f4ef-4e03-a500-64923d854fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"
Oct 30 20:19:47.369981 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.369959    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-qr6wl\" (UID: \"ceb0f5c7-f4ef-4e03-a500-64923d854fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"
Oct 30 20:19:47.371790 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.371765    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-qr6wl\" (UID: \"ceb0f5c7-f4ef-4e03-a500-64923d854fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"
Oct 30 20:19:47.382292 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.382270    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxjhb\" (UniqueName: \"kubernetes.io/projected/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-kube-api-access-mxjhb\") pod \"maas-api-revoke-sub-79805ca9-qr6wl\" (UID: \"ceb0f5c7-f4ef-4e03-a500-64923d854fe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"
Oct 30 20:19:47.546704 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.546622    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"
Oct 30 20:19:47.826020 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:47.825990    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"]
Oct 30 20:19:47.828183 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:19:47.828156    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb0f5c7_f4ef_4e03_a500_64923d854fe9.slice/crio-ce9f4bad912a7f5a393f6378b926a02c3357aa9fbebde9f2260f62009e3cb032 WatchSource:0}: Error finding container ce9f4bad912a7f5a393f6378b926a02c3357aa9fbebde9f2260f62009e3cb032: Status 404 returned error can't find the container with id ce9f4bad912a7f5a393f6378b926a02c3357aa9fbebde9f2260f62009e3cb032
Oct 30 20:19:48.610251 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:48.610213    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl" event={"ID":"ceb0f5c7-f4ef-4e03-a500-64923d854fe9","Type":"ContainerStarted","Data":"a9e3c4111d72745d6f7ac7840b76dd0a46ffec26be6baf9f54a72f36f4c1f370"}
Oct 30 20:19:48.610725 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:48.610294    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl" event={"ID":"ceb0f5c7-f4ef-4e03-a500-64923d854fe9","Type":"ContainerStarted","Data":"ce9f4bad912a7f5a393f6378b926a02c3357aa9fbebde9f2260f62009e3cb032"}
Oct 30 20:19:48.629577 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:48.629529    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl" podStartSLOduration=1.6295149389999999 podStartE2EDuration="1.629514939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:48.626507041 +0000 UTC m=+2988.519121366" watchObservedRunningTime="2026-10-30 20:19:48.629514939 +0000 UTC m=+2988.522129262"
Oct 30 20:19:49.618990 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:49.618955    4823 generic.go:356] "Generic (PLEG): container finished" podID="ceb0f5c7-f4ef-4e03-a500-64923d854fe9" containerID="a9e3c4111d72745d6f7ac7840b76dd0a46ffec26be6baf9f54a72f36f4c1f370" exitCode=0
Oct 30 20:19:49.619410 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:49.619035    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl" event={"ID":"ceb0f5c7-f4ef-4e03-a500-64923d854fe9","Type":"ContainerDied","Data":"a9e3c4111d72745d6f7ac7840b76dd0a46ffec26be6baf9f54a72f36f4c1f370"}
Oct 30 20:19:50.772643 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:50.772617    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"
Oct 30 20:19:50.804787 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:50.804754    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-kube-api-access-mxjhb\" (UniqueName: \"kubernetes.io/projected/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-kube-api-access-mxjhb\") pod \"ceb0f5c7-f4ef-4e03-a500-64923d854fe9\" (UID: \"ceb0f5c7-f4ef-4e03-a500-64923d854fe9\") "
Oct 30 20:19:50.804951 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:50.804812    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-maas-api-xks-ca\") pod \"ceb0f5c7-f4ef-4e03-a500-64923d854fe9\" (UID: \"ceb0f5c7-f4ef-4e03-a500-64923d854fe9\") "
Oct 30 20:19:50.805069 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:50.805050    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-maas-api-openshift-ca\") pod \"ceb0f5c7-f4ef-4e03-a500-64923d854fe9\" (UID: \"ceb0f5c7-f4ef-4e03-a500-64923d854fe9\") "
Oct 30 20:19:50.805357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:50.805332    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-maas-api-openshift-ca" pod "ceb0f5c7-f4ef-4e03-a500-64923d854fe9" (UID: "ceb0f5c7-f4ef-4e03-a500-64923d854fe9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:50.805442 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:50.805416    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:50.807075 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:50.807051    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-maas-api-xks-ca" pod "ceb0f5c7-f4ef-4e03-a500-64923d854fe9" (UID: "ceb0f5c7-f4ef-4e03-a500-64923d854fe9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:50.807212 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:50.807195    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-kube-api-access-mxjhb" pod "ceb0f5c7-f4ef-4e03-a500-64923d854fe9" (UID: "ceb0f5c7-f4ef-4e03-a500-64923d854fe9"). InnerVolumeSpecName "kube-api-access-mxjhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:50.906357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:50.906279    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxjhb\" (UniqueName: \"kubernetes.io/projected/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-kube-api-access-mxjhb\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:50.906357 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:50.906309    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceb0f5c7-f4ef-4e03-a500-64923d854fe9-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:51.631519 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:51.631492    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"
Oct 30 20:19:51.631709 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:51.631488    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl" event={"ID":"ceb0f5c7-f4ef-4e03-a500-64923d854fe9","Type":"ContainerDied","Data":"ce9f4bad912a7f5a393f6378b926a02c3357aa9fbebde9f2260f62009e3cb032"}
Oct 30 20:19:51.631709 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:51.631618    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce9f4bad912a7f5a393f6378b926a02c3357aa9fbebde9f2260f62009e3cb032"
Oct 30 20:19:58.216019 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:19:58.215984    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:20:01.645684 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:01.645601    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:20:01.651837 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:01.651806    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:20:02.044451 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:20:02.044357    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:20:02.392661 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:02.392623    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:20:04.293697 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:04.293669    4823 scope.go:122] "RemoveContainer" containerID="8bd35d6343652e88c6d0b90c5d3ce84e77d73badd70f2b233f50597bd1e3744d"
Oct 30 20:20:05.090453 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.090412    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"]
Oct 30 20:20:05.091011 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.090990    4823 cpu_manager.go:414] "containerMap: removing container" podUID="ceb0f5c7-f4ef-4e03-a500-64923d854fe9" containerName="revoke-keys"
Oct 30 20:20:05.091011 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.091011    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceb0f5c7-f4ef-4e03-a500-64923d854fe9" containerName="revoke-keys"
Oct 30 20:20:05.091190 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.091135    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceb0f5c7-f4ef-4e03-a500-64923d854fe9" containerName="revoke-keys"
Oct 30 20:20:05.095714 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.095693    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"
Oct 30 20:20:05.099684 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.099659    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:20:05.100081 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.099997    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:20:05.100197 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.100075    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:20:05.100309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.100284    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:20:05.103972 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.103919    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"]
Oct 30 20:20:05.163448 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.163416    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzxt9\" (UniqueName: \"kubernetes.io/projected/9cdc026a-a852-442c-9ce5-b0affb671d30-kube-api-access-tzxt9\") pod \"maas-api-revoke-sub-0b7e8edb-mvxkj\" (UID: \"9cdc026a-a852-442c-9ce5-b0affb671d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"
Oct 30 20:20:05.163617 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.163537    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cdc026a-a852-442c-9ce5-b0affb671d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mvxkj\" (UID: \"9cdc026a-a852-442c-9ce5-b0affb671d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"
Oct 30 20:20:05.163617 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.163585    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cdc026a-a852-442c-9ce5-b0affb671d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mvxkj\" (UID: \"9cdc026a-a852-442c-9ce5-b0affb671d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"
Oct 30 20:20:05.265022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.264987    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cdc026a-a852-442c-9ce5-b0affb671d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mvxkj\" (UID: \"9cdc026a-a852-442c-9ce5-b0affb671d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"
Oct 30 20:20:05.265213 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.265046    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cdc026a-a852-442c-9ce5-b0affb671d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mvxkj\" (UID: \"9cdc026a-a852-442c-9ce5-b0affb671d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"
Oct 30 20:20:05.265213 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.265150    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzxt9\" (UniqueName: \"kubernetes.io/projected/9cdc026a-a852-442c-9ce5-b0affb671d30-kube-api-access-tzxt9\") pod \"maas-api-revoke-sub-0b7e8edb-mvxkj\" (UID: \"9cdc026a-a852-442c-9ce5-b0affb671d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"
Oct 30 20:20:05.265707 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.265680    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cdc026a-a852-442c-9ce5-b0affb671d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mvxkj\" (UID: \"9cdc026a-a852-442c-9ce5-b0affb671d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"
Oct 30 20:20:05.267486 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.267457    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cdc026a-a852-442c-9ce5-b0affb671d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mvxkj\" (UID: \"9cdc026a-a852-442c-9ce5-b0affb671d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"
Oct 30 20:20:05.277093 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.277073    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzxt9\" (UniqueName: \"kubernetes.io/projected/9cdc026a-a852-442c-9ce5-b0affb671d30-kube-api-access-tzxt9\") pod \"maas-api-revoke-sub-0b7e8edb-mvxkj\" (UID: \"9cdc026a-a852-442c-9ce5-b0affb671d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"
Oct 30 20:20:05.410991 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.410957    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"
Oct 30 20:20:05.694031 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.694005    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"]
Oct 30 20:20:05.696183 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:20:05.696158    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdc026a_a852_442c_9ce5_b0affb671d30.slice/crio-d066929cb89b6c82c9b64adb6f1686da80d938fe20e0aea141428b5a7fad3d7b WatchSource:0}: Error finding container d066929cb89b6c82c9b64adb6f1686da80d938fe20e0aea141428b5a7fad3d7b: Status 404 returned error can't find the container with id d066929cb89b6c82c9b64adb6f1686da80d938fe20e0aea141428b5a7fad3d7b
Oct 30 20:20:05.752044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:05.752007    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj" event={"ID":"9cdc026a-a852-442c-9ce5-b0affb671d30","Type":"ContainerStarted","Data":"d066929cb89b6c82c9b64adb6f1686da80d938fe20e0aea141428b5a7fad3d7b"}
Oct 30 20:20:06.758685 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:06.758591    4823 generic.go:356] "Generic (PLEG): container finished" podID="9cdc026a-a852-442c-9ce5-b0affb671d30" containerID="26ba9e4b3fbf2ca08127de96e032081a7746ca40b9c3b6f7ec4e9c5f3c6912ba" exitCode=0
Oct 30 20:20:06.758685 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:06.758629    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj" event={"ID":"9cdc026a-a852-442c-9ce5-b0affb671d30","Type":"ContainerDied","Data":"26ba9e4b3fbf2ca08127de96e032081a7746ca40b9c3b6f7ec4e9c5f3c6912ba"}
Oct 30 20:20:07.919501 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:07.919477    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"
Oct 30 20:20:07.993675 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:07.993638    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cdc026a-a852-442c-9ce5-b0affb671d30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cdc026a-a852-442c-9ce5-b0affb671d30-maas-api-openshift-ca\") pod \"9cdc026a-a852-442c-9ce5-b0affb671d30\" (UID: \"9cdc026a-a852-442c-9ce5-b0affb671d30\") "
Oct 30 20:20:07.993855 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:07.993701    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cdc026a-a852-442c-9ce5-b0affb671d30-kube-api-access-tzxt9\" (UniqueName: \"kubernetes.io/projected/9cdc026a-a852-442c-9ce5-b0affb671d30-kube-api-access-tzxt9\") pod \"9cdc026a-a852-442c-9ce5-b0affb671d30\" (UID: \"9cdc026a-a852-442c-9ce5-b0affb671d30\") "
Oct 30 20:20:07.993855 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:07.993823    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cdc026a-a852-442c-9ce5-b0affb671d30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cdc026a-a852-442c-9ce5-b0affb671d30-maas-api-xks-ca\") pod \"9cdc026a-a852-442c-9ce5-b0affb671d30\" (UID: \"9cdc026a-a852-442c-9ce5-b0affb671d30\") "
Oct 30 20:20:07.994124 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:07.994097    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cdc026a-a852-442c-9ce5-b0affb671d30-maas-api-openshift-ca" pod "9cdc026a-a852-442c-9ce5-b0affb671d30" (UID: "9cdc026a-a852-442c-9ce5-b0affb671d30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:07.994306 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:07.994284    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cdc026a-a852-442c-9ce5-b0affb671d30-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:07.996047 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:07.996010    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdc026a-a852-442c-9ce5-b0affb671d30-kube-api-access-tzxt9" pod "9cdc026a-a852-442c-9ce5-b0affb671d30" (UID: "9cdc026a-a852-442c-9ce5-b0affb671d30"). InnerVolumeSpecName "kube-api-access-tzxt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:07.996166 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:07.996138    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cdc026a-a852-442c-9ce5-b0affb671d30-maas-api-xks-ca" pod "9cdc026a-a852-442c-9ce5-b0affb671d30" (UID: "9cdc026a-a852-442c-9ce5-b0affb671d30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:08.095060 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:08.094968    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzxt9\" (UniqueName: \"kubernetes.io/projected/9cdc026a-a852-442c-9ce5-b0affb671d30-kube-api-access-tzxt9\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.095060 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:08.095007    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cdc026a-a852-442c-9ce5-b0affb671d30-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.771654 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:08.771619    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"
Oct 30 20:20:08.772119 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:08.771622    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj" event={"ID":"9cdc026a-a852-442c-9ce5-b0affb671d30","Type":"ContainerDied","Data":"d066929cb89b6c82c9b64adb6f1686da80d938fe20e0aea141428b5a7fad3d7b"}
Oct 30 20:20:08.772119 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:08.771756    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d066929cb89b6c82c9b64adb6f1686da80d938fe20e0aea141428b5a7fad3d7b"
Oct 30 20:20:15.870639 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:15.870560    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"]
Oct 30 20:20:15.871174 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:15.871065    4823 cpu_manager.go:414] "containerMap: removing container" podUID="9cdc026a-a852-442c-9ce5-b0affb671d30" containerName="revoke-keys"
Oct 30 20:20:15.871174 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:15.871077    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cdc026a-a852-442c-9ce5-b0affb671d30" containerName="revoke-keys"
Oct 30 20:20:15.871174 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:15.871144    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cdc026a-a852-442c-9ce5-b0affb671d30" containerName="revoke-keys"
Oct 30 20:20:15.874656 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:15.874634    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"
Oct 30 20:20:15.878004 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:15.877951    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:20:15.878171 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:15.878141    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:20:15.878243 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:15.878201    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:20:15.878299 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:15.878261    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:20:15.887582 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:15.887543    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"]
Oct 30 20:20:15.974079 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:15.974045    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5719fb1-3afb-4c07-bee0-585f2a6701cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c8m4p\" (UID: \"c5719fb1-3afb-4c07-bee0-585f2a6701cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"
Oct 30 20:20:15.974267 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:15.974095    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9kfb\" (UniqueName: \"kubernetes.io/projected/c5719fb1-3afb-4c07-bee0-585f2a6701cb-kube-api-access-g9kfb\") pod \"maas-api-revoke-sub-3d4156f1-c8m4p\" (UID: \"c5719fb1-3afb-4c07-bee0-585f2a6701cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"
Oct 30 20:20:15.974267 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:15.974217    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5719fb1-3afb-4c07-bee0-585f2a6701cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c8m4p\" (UID: \"c5719fb1-3afb-4c07-bee0-585f2a6701cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"
Oct 30 20:20:15.995954 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:15.995920    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:20:16.075206 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:16.075174    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5719fb1-3afb-4c07-bee0-585f2a6701cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c8m4p\" (UID: \"c5719fb1-3afb-4c07-bee0-585f2a6701cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"
Oct 30 20:20:16.075410 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:16.075215    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9kfb\" (UniqueName: \"kubernetes.io/projected/c5719fb1-3afb-4c07-bee0-585f2a6701cb-kube-api-access-g9kfb\") pod \"maas-api-revoke-sub-3d4156f1-c8m4p\" (UID: \"c5719fb1-3afb-4c07-bee0-585f2a6701cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"
Oct 30 20:20:16.075410 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:16.075273    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5719fb1-3afb-4c07-bee0-585f2a6701cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c8m4p\" (UID: \"c5719fb1-3afb-4c07-bee0-585f2a6701cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"
Oct 30 20:20:16.075857 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:16.075830    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5719fb1-3afb-4c07-bee0-585f2a6701cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c8m4p\" (UID: \"c5719fb1-3afb-4c07-bee0-585f2a6701cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"
Oct 30 20:20:16.077837 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:16.077814    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5719fb1-3afb-4c07-bee0-585f2a6701cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c8m4p\" (UID: \"c5719fb1-3afb-4c07-bee0-585f2a6701cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"
Oct 30 20:20:16.095294 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:16.095261    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9kfb\" (UniqueName: \"kubernetes.io/projected/c5719fb1-3afb-4c07-bee0-585f2a6701cb-kube-api-access-g9kfb\") pod \"maas-api-revoke-sub-3d4156f1-c8m4p\" (UID: \"c5719fb1-3afb-4c07-bee0-585f2a6701cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"
Oct 30 20:20:16.199084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:16.199009    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"
Oct 30 20:20:16.489347 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:16.489318    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"]
Oct 30 20:20:16.491752 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:20:16.491724    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5719fb1_3afb_4c07_bee0_585f2a6701cb.slice/crio-dcd6984039ed0f9f45c44ae8bb43cf7ed9fdcc32433e8181b29ade81c49f1498 WatchSource:0}: Error finding container dcd6984039ed0f9f45c44ae8bb43cf7ed9fdcc32433e8181b29ade81c49f1498: Status 404 returned error can't find the container with id dcd6984039ed0f9f45c44ae8bb43cf7ed9fdcc32433e8181b29ade81c49f1498
Oct 30 20:20:16.819795 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:16.819761    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p" event={"ID":"c5719fb1-3afb-4c07-bee0-585f2a6701cb","Type":"ContainerStarted","Data":"50c1cfaf1e69c38f60f13c7dbb89a5589f8e458968f6fcd0e578a05331f69f71"}
Oct 30 20:20:16.819991 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:16.819818    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p" event={"ID":"c5719fb1-3afb-4c07-bee0-585f2a6701cb","Type":"ContainerStarted","Data":"dcd6984039ed0f9f45c44ae8bb43cf7ed9fdcc32433e8181b29ade81c49f1498"}
Oct 30 20:20:17.827828 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:17.827793    4823 generic.go:356] "Generic (PLEG): container finished" podID="c5719fb1-3afb-4c07-bee0-585f2a6701cb" containerID="50c1cfaf1e69c38f60f13c7dbb89a5589f8e458968f6fcd0e578a05331f69f71" exitCode=0
Oct 30 20:20:17.828264 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:17.827892    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p" event={"ID":"c5719fb1-3afb-4c07-bee0-585f2a6701cb","Type":"ContainerDied","Data":"50c1cfaf1e69c38f60f13c7dbb89a5589f8e458968f6fcd0e578a05331f69f71"}
Oct 30 20:20:18.981939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:18.981914    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"
Oct 30 20:20:19.112018 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:19.111985    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5719fb1-3afb-4c07-bee0-585f2a6701cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5719fb1-3afb-4c07-bee0-585f2a6701cb-maas-api-openshift-ca\") pod \"c5719fb1-3afb-4c07-bee0-585f2a6701cb\" (UID: \"c5719fb1-3afb-4c07-bee0-585f2a6701cb\") "
Oct 30 20:20:19.112220 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:19.112101    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5719fb1-3afb-4c07-bee0-585f2a6701cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5719fb1-3afb-4c07-bee0-585f2a6701cb-maas-api-xks-ca\") pod \"c5719fb1-3afb-4c07-bee0-585f2a6701cb\" (UID: \"c5719fb1-3afb-4c07-bee0-585f2a6701cb\") "
Oct 30 20:20:19.112220 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:19.112174    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5719fb1-3afb-4c07-bee0-585f2a6701cb-kube-api-access-g9kfb\" (UniqueName: \"kubernetes.io/projected/c5719fb1-3afb-4c07-bee0-585f2a6701cb-kube-api-access-g9kfb\") pod \"c5719fb1-3afb-4c07-bee0-585f2a6701cb\" (UID: \"c5719fb1-3afb-4c07-bee0-585f2a6701cb\") "
Oct 30 20:20:19.112437 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:19.112409    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5719fb1-3afb-4c07-bee0-585f2a6701cb-maas-api-openshift-ca" pod "c5719fb1-3afb-4c07-bee0-585f2a6701cb" (UID: "c5719fb1-3afb-4c07-bee0-585f2a6701cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:19.112597 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:19.112578    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5719fb1-3afb-4c07-bee0-585f2a6701cb-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:19.114392 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:19.114364    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5719fb1-3afb-4c07-bee0-585f2a6701cb-maas-api-xks-ca" pod "c5719fb1-3afb-4c07-bee0-585f2a6701cb" (UID: "c5719fb1-3afb-4c07-bee0-585f2a6701cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:19.114506 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:19.114412    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5719fb1-3afb-4c07-bee0-585f2a6701cb-kube-api-access-g9kfb" pod "c5719fb1-3afb-4c07-bee0-585f2a6701cb" (UID: "c5719fb1-3afb-4c07-bee0-585f2a6701cb"). InnerVolumeSpecName "kube-api-access-g9kfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:19.214086 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:19.214050    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9kfb\" (UniqueName: \"kubernetes.io/projected/c5719fb1-3afb-4c07-bee0-585f2a6701cb-kube-api-access-g9kfb\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:19.214086 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:19.214083    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5719fb1-3afb-4c07-bee0-585f2a6701cb-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:19.839962 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:19.839931    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"
Oct 30 20:20:19.840139 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:19.839963    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p" event={"ID":"c5719fb1-3afb-4c07-bee0-585f2a6701cb","Type":"ContainerDied","Data":"dcd6984039ed0f9f45c44ae8bb43cf7ed9fdcc32433e8181b29ade81c49f1498"}
Oct 30 20:20:19.840139 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:19.840044    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcd6984039ed0f9f45c44ae8bb43cf7ed9fdcc32433e8181b29ade81c49f1498"
Oct 30 20:20:26.733150 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:26.733116    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:20:31.683988 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:31.683937    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:20:32.048033 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:20:32.047950    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:20:37.228294 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.228259    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"]
Oct 30 20:20:37.237633 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.237604    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c8m4p"]
Oct 30 20:20:37.257147 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.257115    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"]
Oct 30 20:20:37.257651 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.257636    4823 cpu_manager.go:414] "containerMap: removing container" podUID="c5719fb1-3afb-4c07-bee0-585f2a6701cb" containerName="revoke-keys"
Oct 30 20:20:37.257726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.257652    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5719fb1-3afb-4c07-bee0-585f2a6701cb" containerName="revoke-keys"
Oct 30 20:20:37.257786 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.257751    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5719fb1-3afb-4c07-bee0-585f2a6701cb" containerName="revoke-keys"
Oct 30 20:20:37.261127 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.261108    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"
Oct 30 20:20:37.266683 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.266657    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:20:37.266826 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.266741    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:20:37.266954 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.266845    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:20:37.267628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.267604    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:20:37.272049 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.272029    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"]
Oct 30 20:20:37.335504 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.335466    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-59twx\" (UID: \"ecefb34c-9b47-4d8a-a5f6-22c5908ac135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"
Oct 30 20:20:37.335685 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.335528    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgk99\" (UniqueName: \"kubernetes.io/projected/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-kube-api-access-dgk99\") pod \"maas-api-revoke-sub-3d4156f1-59twx\" (UID: \"ecefb34c-9b47-4d8a-a5f6-22c5908ac135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"
Oct 30 20:20:37.335755 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.335731    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-59twx\" (UID: \"ecefb34c-9b47-4d8a-a5f6-22c5908ac135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"
Oct 30 20:20:37.436766 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.436732    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-59twx\" (UID: \"ecefb34c-9b47-4d8a-a5f6-22c5908ac135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"
Oct 30 20:20:37.437022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.436793    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgk99\" (UniqueName: \"kubernetes.io/projected/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-kube-api-access-dgk99\") pod \"maas-api-revoke-sub-3d4156f1-59twx\" (UID: \"ecefb34c-9b47-4d8a-a5f6-22c5908ac135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"
Oct 30 20:20:37.437022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.436835    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-59twx\" (UID: \"ecefb34c-9b47-4d8a-a5f6-22c5908ac135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"
Oct 30 20:20:37.437559 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.437536    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-59twx\" (UID: \"ecefb34c-9b47-4d8a-a5f6-22c5908ac135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"
Oct 30 20:20:37.439203 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.439177    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-59twx\" (UID: \"ecefb34c-9b47-4d8a-a5f6-22c5908ac135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"
Oct 30 20:20:37.450058 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.450036    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgk99\" (UniqueName: \"kubernetes.io/projected/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-kube-api-access-dgk99\") pod \"maas-api-revoke-sub-3d4156f1-59twx\" (UID: \"ecefb34c-9b47-4d8a-a5f6-22c5908ac135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"
Oct 30 20:20:37.576736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.576653    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"
Oct 30 20:20:37.855756 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.855728    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"]
Oct 30 20:20:37.858054 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:20:37.858027    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecefb34c_9b47_4d8a_a5f6_22c5908ac135.slice/crio-d2518d8ed4df5b3a6271fbef5b7a5868fdd8e3f5c536f2e39ba339848ca5fb00 WatchSource:0}: Error finding container d2518d8ed4df5b3a6271fbef5b7a5868fdd8e3f5c536f2e39ba339848ca5fb00: Status 404 returned error can't find the container with id d2518d8ed4df5b3a6271fbef5b7a5868fdd8e3f5c536f2e39ba339848ca5fb00
Oct 30 20:20:37.983410 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:37.983378    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx" event={"ID":"ecefb34c-9b47-4d8a-a5f6-22c5908ac135","Type":"ContainerStarted","Data":"d2518d8ed4df5b3a6271fbef5b7a5868fdd8e3f5c536f2e39ba339848ca5fb00"}
Oct 30 20:20:38.278006 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:38.277969    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5719fb1-3afb-4c07-bee0-585f2a6701cb" path="/var/lib/kubelet/pods/c5719fb1-3afb-4c07-bee0-585f2a6701cb/volumes"
Oct 30 20:20:38.990154 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:38.990112    4823 generic.go:356] "Generic (PLEG): container finished" podID="ecefb34c-9b47-4d8a-a5f6-22c5908ac135" containerID="f93484268ef08a285fa046f4ae4c863d289d181a315d4843a388ca4b8fd15ace" exitCode=0
Oct 30 20:20:38.990346 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:38.990206    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx" event={"ID":"ecefb34c-9b47-4d8a-a5f6-22c5908ac135","Type":"ContainerDied","Data":"f93484268ef08a285fa046f4ae4c863d289d181a315d4843a388ca4b8fd15ace"}
Oct 30 20:20:40.140335 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:40.140309    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"
Oct 30 20:20:40.273748 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:40.273661    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-kube-api-access-dgk99\" (UniqueName: \"kubernetes.io/projected/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-kube-api-access-dgk99\") pod \"ecefb34c-9b47-4d8a-a5f6-22c5908ac135\" (UID: \"ecefb34c-9b47-4d8a-a5f6-22c5908ac135\") "
Oct 30 20:20:40.273927 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:40.273874    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-maas-api-openshift-ca\") pod \"ecefb34c-9b47-4d8a-a5f6-22c5908ac135\" (UID: \"ecefb34c-9b47-4d8a-a5f6-22c5908ac135\") "
Oct 30 20:20:40.273973 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:40.273930    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-maas-api-xks-ca\") pod \"ecefb34c-9b47-4d8a-a5f6-22c5908ac135\" (UID: \"ecefb34c-9b47-4d8a-a5f6-22c5908ac135\") "
Oct 30 20:20:40.274252 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:40.274212    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-maas-api-openshift-ca" pod "ecefb34c-9b47-4d8a-a5f6-22c5908ac135" (UID: "ecefb34c-9b47-4d8a-a5f6-22c5908ac135"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:40.274834 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:40.274801    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:40.276652 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:40.276613    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-kube-api-access-dgk99" pod "ecefb34c-9b47-4d8a-a5f6-22c5908ac135" (UID: "ecefb34c-9b47-4d8a-a5f6-22c5908ac135"). InnerVolumeSpecName "kube-api-access-dgk99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:40.285514 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:40.285481    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-maas-api-xks-ca" pod "ecefb34c-9b47-4d8a-a5f6-22c5908ac135" (UID: "ecefb34c-9b47-4d8a-a5f6-22c5908ac135"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:40.376388 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:40.376350    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgk99\" (UniqueName: \"kubernetes.io/projected/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-kube-api-access-dgk99\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:40.376388 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:40.376386    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecefb34c-9b47-4d8a-a5f6-22c5908ac135-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:20:41.003362 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:41.003325    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx" event={"ID":"ecefb34c-9b47-4d8a-a5f6-22c5908ac135","Type":"ContainerDied","Data":"d2518d8ed4df5b3a6271fbef5b7a5868fdd8e3f5c536f2e39ba339848ca5fb00"}
Oct 30 20:20:41.003544 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:41.003364    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"
Oct 30 20:20:41.003544 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:41.003381    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2518d8ed4df5b3a6271fbef5b7a5868fdd8e3f5c536f2e39ba339848ca5fb00"
Oct 30 20:20:48.186157 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:48.186113    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:20:51.620085 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:51.620038    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:20:58.031680 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:58.031637    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"]
Oct 30 20:20:58.037372 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:58.037344    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df9277f6-6qwv7"]
Oct 30 20:20:58.278536 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:58.278499    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da55fc12-f908-4bbe-9283-7446b1556be9" path="/var/lib/kubelet/pods/da55fc12-f908-4bbe-9283-7446b1556be9/volumes"
Oct 30 20:20:59.015281 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:20:59.015247    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:21:02.040414 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:21:02.040370    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:21:06.358816 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.358773    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"]
Oct 30 20:21:06.359312 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.359295    4823 cpu_manager.go:414] "containerMap: removing container" podUID="ecefb34c-9b47-4d8a-a5f6-22c5908ac135" containerName="revoke-keys"
Oct 30 20:21:06.359379 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.359313    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecefb34c-9b47-4d8a-a5f6-22c5908ac135" containerName="revoke-keys"
Oct 30 20:21:06.359428 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.359423    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecefb34c-9b47-4d8a-a5f6-22c5908ac135" containerName="revoke-keys"
Oct 30 20:21:06.362713 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.362693    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"
Oct 30 20:21:06.377095 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.377071    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:21:06.377429 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.377406    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:21:06.377537 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.377489    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:21:06.379840 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.379815    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:21:06.380932 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.380914    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"]
Oct 30 20:21:06.467184 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.467148    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdk7\" (UniqueName: \"kubernetes.io/projected/dcff8490-90ba-4219-86e5-042c9b03de29-kube-api-access-gjdk7\") pod \"maas-api-revoke-sub-e2fc9c5b-n2x2f\" (UID: \"dcff8490-90ba-4219-86e5-042c9b03de29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"
Oct 30 20:21:06.467376 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.467190    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcff8490-90ba-4219-86e5-042c9b03de29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-n2x2f\" (UID: \"dcff8490-90ba-4219-86e5-042c9b03de29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"
Oct 30 20:21:06.467376 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.467219    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcff8490-90ba-4219-86e5-042c9b03de29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-n2x2f\" (UID: \"dcff8490-90ba-4219-86e5-042c9b03de29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"
Oct 30 20:21:06.568124 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.568072    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdk7\" (UniqueName: \"kubernetes.io/projected/dcff8490-90ba-4219-86e5-042c9b03de29-kube-api-access-gjdk7\") pod \"maas-api-revoke-sub-e2fc9c5b-n2x2f\" (UID: \"dcff8490-90ba-4219-86e5-042c9b03de29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"
Oct 30 20:21:06.568124 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.568128    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcff8490-90ba-4219-86e5-042c9b03de29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-n2x2f\" (UID: \"dcff8490-90ba-4219-86e5-042c9b03de29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"
Oct 30 20:21:06.568404 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.568147    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcff8490-90ba-4219-86e5-042c9b03de29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-n2x2f\" (UID: \"dcff8490-90ba-4219-86e5-042c9b03de29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"
Oct 30 20:21:06.568844 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.568818    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcff8490-90ba-4219-86e5-042c9b03de29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-n2x2f\" (UID: \"dcff8490-90ba-4219-86e5-042c9b03de29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"
Oct 30 20:21:06.570595 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.570571    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcff8490-90ba-4219-86e5-042c9b03de29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-n2x2f\" (UID: \"dcff8490-90ba-4219-86e5-042c9b03de29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"
Oct 30 20:21:06.580574 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.580552    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdk7\" (UniqueName: \"kubernetes.io/projected/dcff8490-90ba-4219-86e5-042c9b03de29-kube-api-access-gjdk7\") pod \"maas-api-revoke-sub-e2fc9c5b-n2x2f\" (UID: \"dcff8490-90ba-4219-86e5-042c9b03de29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"
Oct 30 20:21:06.676148 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.676082    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"
Oct 30 20:21:06.957104 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:06.957080    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"]
Oct 30 20:21:06.959533 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:21:06.959505    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcff8490_90ba_4219_86e5_042c9b03de29.slice/crio-f06e80fffe824cea55b219c9653135c8a8566241c2f931fb53bc9bf755ec7bea WatchSource:0}: Error finding container f06e80fffe824cea55b219c9653135c8a8566241c2f931fb53bc9bf755ec7bea: Status 404 returned error can't find the container with id f06e80fffe824cea55b219c9653135c8a8566241c2f931fb53bc9bf755ec7bea
Oct 30 20:21:07.161196 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:07.161161    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f" event={"ID":"dcff8490-90ba-4219-86e5-042c9b03de29","Type":"ContainerStarted","Data":"f06e80fffe824cea55b219c9653135c8a8566241c2f931fb53bc9bf755ec7bea"}
Oct 30 20:21:08.168773 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:08.168740    4823 generic.go:356] "Generic (PLEG): container finished" podID="dcff8490-90ba-4219-86e5-042c9b03de29" containerID="29bd4f2ee5197d3c9fe85a00372257bb13ad098a5e50999b197f7feb77b0adc0" exitCode=0
Oct 30 20:21:08.169184 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:08.168821    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f" event={"ID":"dcff8490-90ba-4219-86e5-042c9b03de29","Type":"ContainerDied","Data":"29bd4f2ee5197d3c9fe85a00372257bb13ad098a5e50999b197f7feb77b0adc0"}
Oct 30 20:21:09.315820 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:09.315795    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"
Oct 30 20:21:09.395762 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:09.395730    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcff8490-90ba-4219-86e5-042c9b03de29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcff8490-90ba-4219-86e5-042c9b03de29-maas-api-xks-ca\") pod \"dcff8490-90ba-4219-86e5-042c9b03de29\" (UID: \"dcff8490-90ba-4219-86e5-042c9b03de29\") "
Oct 30 20:21:09.395939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:09.395781    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcff8490-90ba-4219-86e5-042c9b03de29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcff8490-90ba-4219-86e5-042c9b03de29-maas-api-openshift-ca\") pod \"dcff8490-90ba-4219-86e5-042c9b03de29\" (UID: \"dcff8490-90ba-4219-86e5-042c9b03de29\") "
Oct 30 20:21:09.395939 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:09.395837    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcff8490-90ba-4219-86e5-042c9b03de29-kube-api-access-gjdk7\" (UniqueName: \"kubernetes.io/projected/dcff8490-90ba-4219-86e5-042c9b03de29-kube-api-access-gjdk7\") pod \"dcff8490-90ba-4219-86e5-042c9b03de29\" (UID: \"dcff8490-90ba-4219-86e5-042c9b03de29\") "
Oct 30 20:21:09.396215 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:09.396183    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcff8490-90ba-4219-86e5-042c9b03de29-maas-api-openshift-ca" pod "dcff8490-90ba-4219-86e5-042c9b03de29" (UID: "dcff8490-90ba-4219-86e5-042c9b03de29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:09.398026 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:09.397997    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcff8490-90ba-4219-86e5-042c9b03de29-kube-api-access-gjdk7" pod "dcff8490-90ba-4219-86e5-042c9b03de29" (UID: "dcff8490-90ba-4219-86e5-042c9b03de29"). InnerVolumeSpecName "kube-api-access-gjdk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:09.398026 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:09.398011    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcff8490-90ba-4219-86e5-042c9b03de29-maas-api-xks-ca" pod "dcff8490-90ba-4219-86e5-042c9b03de29" (UID: "dcff8490-90ba-4219-86e5-042c9b03de29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:09.496952 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:09.496845    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcff8490-90ba-4219-86e5-042c9b03de29-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:09.496952 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:09.496906    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcff8490-90ba-4219-86e5-042c9b03de29-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:09.496952 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:09.496917    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjdk7\" (UniqueName: \"kubernetes.io/projected/dcff8490-90ba-4219-86e5-042c9b03de29-kube-api-access-gjdk7\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:10.181640 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:10.181607    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"
Oct 30 20:21:10.181826 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:10.181631    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f" event={"ID":"dcff8490-90ba-4219-86e5-042c9b03de29","Type":"ContainerDied","Data":"f06e80fffe824cea55b219c9653135c8a8566241c2f931fb53bc9bf755ec7bea"}
Oct 30 20:21:10.181826 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:10.181683    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f06e80fffe824cea55b219c9653135c8a8566241c2f931fb53bc9bf755ec7bea"
Oct 30 20:21:16.648044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.648008    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"]
Oct 30 20:21:16.648699 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.648677    4823 cpu_manager.go:414] "containerMap: removing container" podUID="dcff8490-90ba-4219-86e5-042c9b03de29" containerName="revoke-keys"
Oct 30 20:21:16.648806 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.648702    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcff8490-90ba-4219-86e5-042c9b03de29" containerName="revoke-keys"
Oct 30 20:21:16.648888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.648831    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcff8490-90ba-4219-86e5-042c9b03de29" containerName="revoke-keys"
Oct 30 20:21:16.652229 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.652209    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"
Oct 30 20:21:16.657227 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.657207    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:21:16.660508 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.660490    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:21:16.665181 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.663043    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:21:16.665181 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.663273    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:21:16.665181 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.665130    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"]
Oct 30 20:21:16.772414 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.772380    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/158e8d97-91c4-4196-b366-16ee81961f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-8b66z\" (UID: \"158e8d97-91c4-4196-b366-16ee81961f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"
Oct 30 20:21:16.772585 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.772494    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5gbl\" (UniqueName: \"kubernetes.io/projected/158e8d97-91c4-4196-b366-16ee81961f6d-kube-api-access-c5gbl\") pod \"maas-api-revoke-sub-5a29efb3-8b66z\" (UID: \"158e8d97-91c4-4196-b366-16ee81961f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"
Oct 30 20:21:16.772585 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.772556    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/158e8d97-91c4-4196-b366-16ee81961f6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-8b66z\" (UID: \"158e8d97-91c4-4196-b366-16ee81961f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"
Oct 30 20:21:16.873306 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.873270    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/158e8d97-91c4-4196-b366-16ee81961f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-8b66z\" (UID: \"158e8d97-91c4-4196-b366-16ee81961f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"
Oct 30 20:21:16.873502 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.873375    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5gbl\" (UniqueName: \"kubernetes.io/projected/158e8d97-91c4-4196-b366-16ee81961f6d-kube-api-access-c5gbl\") pod \"maas-api-revoke-sub-5a29efb3-8b66z\" (UID: \"158e8d97-91c4-4196-b366-16ee81961f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"
Oct 30 20:21:16.873502 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.873407    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/158e8d97-91c4-4196-b366-16ee81961f6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-8b66z\" (UID: \"158e8d97-91c4-4196-b366-16ee81961f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"
Oct 30 20:21:16.874014 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.873990    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/158e8d97-91c4-4196-b366-16ee81961f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-8b66z\" (UID: \"158e8d97-91c4-4196-b366-16ee81961f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"
Oct 30 20:21:16.875833 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.875811    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/158e8d97-91c4-4196-b366-16ee81961f6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-8b66z\" (UID: \"158e8d97-91c4-4196-b366-16ee81961f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"
Oct 30 20:21:16.885768 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.885733    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5gbl\" (UniqueName: \"kubernetes.io/projected/158e8d97-91c4-4196-b366-16ee81961f6d-kube-api-access-c5gbl\") pod \"maas-api-revoke-sub-5a29efb3-8b66z\" (UID: \"158e8d97-91c4-4196-b366-16ee81961f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"
Oct 30 20:21:16.970785 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:16.970721    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"
Oct 30 20:21:17.257802 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:17.257613    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"]
Oct 30 20:21:17.260747 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:21:17.260715    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158e8d97_91c4_4196_b366_16ee81961f6d.slice/crio-019e833ca607bc9140366de93fcefe08b9d0d2a20c033b62a25b613b264aae0d WatchSource:0}: Error finding container 019e833ca607bc9140366de93fcefe08b9d0d2a20c033b62a25b613b264aae0d: Status 404 returned error can't find the container with id 019e833ca607bc9140366de93fcefe08b9d0d2a20c033b62a25b613b264aae0d
Oct 30 20:21:17.494665 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:17.494630    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:21:18.230605 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:18.230567    4823 generic.go:356] "Generic (PLEG): container finished" podID="158e8d97-91c4-4196-b366-16ee81961f6d" containerID="be3c8de72fc64acdd14c1c2c233eef80727683869de0e302906474f33639bdc5" exitCode=0
Oct 30 20:21:18.231055 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:18.230660    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z" event={"ID":"158e8d97-91c4-4196-b366-16ee81961f6d","Type":"ContainerDied","Data":"be3c8de72fc64acdd14c1c2c233eef80727683869de0e302906474f33639bdc5"}
Oct 30 20:21:18.231055 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:18.230731    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z" event={"ID":"158e8d97-91c4-4196-b366-16ee81961f6d","Type":"ContainerStarted","Data":"019e833ca607bc9140366de93fcefe08b9d0d2a20c033b62a25b613b264aae0d"}
Oct 30 20:21:19.630909 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:19.630808    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"
Oct 30 20:21:19.733855 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:19.733235    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158e8d97-91c4-4196-b366-16ee81961f6d-kube-api-access-c5gbl\" (UniqueName: \"kubernetes.io/projected/158e8d97-91c4-4196-b366-16ee81961f6d-kube-api-access-c5gbl\") pod \"158e8d97-91c4-4196-b366-16ee81961f6d\" (UID: \"158e8d97-91c4-4196-b366-16ee81961f6d\") "
Oct 30 20:21:19.733855 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:19.733315    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158e8d97-91c4-4196-b366-16ee81961f6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/158e8d97-91c4-4196-b366-16ee81961f6d-maas-api-openshift-ca\") pod \"158e8d97-91c4-4196-b366-16ee81961f6d\" (UID: \"158e8d97-91c4-4196-b366-16ee81961f6d\") "
Oct 30 20:21:19.733855 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:19.733361    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/158e8d97-91c4-4196-b366-16ee81961f6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/158e8d97-91c4-4196-b366-16ee81961f6d-maas-api-xks-ca\") pod \"158e8d97-91c4-4196-b366-16ee81961f6d\" (UID: \"158e8d97-91c4-4196-b366-16ee81961f6d\") "
Oct 30 20:21:19.734733 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:19.734705    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158e8d97-91c4-4196-b366-16ee81961f6d-maas-api-openshift-ca" pod "158e8d97-91c4-4196-b366-16ee81961f6d" (UID: "158e8d97-91c4-4196-b366-16ee81961f6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:19.737978 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:19.737947    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/158e8d97-91c4-4196-b366-16ee81961f6d-maas-api-xks-ca" pod "158e8d97-91c4-4196-b366-16ee81961f6d" (UID: "158e8d97-91c4-4196-b366-16ee81961f6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:19.739310 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:19.739287    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158e8d97-91c4-4196-b366-16ee81961f6d-kube-api-access-c5gbl" pod "158e8d97-91c4-4196-b366-16ee81961f6d" (UID: "158e8d97-91c4-4196-b366-16ee81961f6d"). InnerVolumeSpecName "kube-api-access-c5gbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:19.835058 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:19.834946    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/158e8d97-91c4-4196-b366-16ee81961f6d-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:19.835058 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:19.834977    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/158e8d97-91c4-4196-b366-16ee81961f6d-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:19.835058 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:19.834987    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5gbl\" (UniqueName: \"kubernetes.io/projected/158e8d97-91c4-4196-b366-16ee81961f6d-kube-api-access-c5gbl\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:21:20.243715 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:20.243684    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"
Oct 30 20:21:20.243929 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:20.243728    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z" event={"ID":"158e8d97-91c4-4196-b366-16ee81961f6d","Type":"ContainerDied","Data":"019e833ca607bc9140366de93fcefe08b9d0d2a20c033b62a25b613b264aae0d"}
Oct 30 20:21:20.243929 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:20.243772    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="019e833ca607bc9140366de93fcefe08b9d0d2a20c033b62a25b613b264aae0d"
Oct 30 20:21:27.603633 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:27.603587    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:21:32.013614 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:21:32.013571    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:21:46.043251 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:46.043169    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"]
Oct 30 20:21:46.048253 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:46.048182    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lf8r9"]
Oct 30 20:21:46.277599 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:21:46.277563    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e15244c-6697-44f9-b284-5538edb35196" path="/var/lib/kubelet/pods/5e15244c-6697-44f9-b284-5538edb35196/volumes"
Oct 30 20:22:02.013923 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:22:02.013881    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:22:04.324624 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:22:04.324592    4823 scope.go:122] "RemoveContainer" containerID="2a41cfd9250979b3d35b2b089d8e393f909eaea821dac1425b28ff42a97eeb30"
Oct 30 20:22:32.014878 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:22:32.014829    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:23:02.014526 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:23:02.014485    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:23:04.346574 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:23:04.346540    4823 scope.go:122] "RemoveContainer" containerID="07732c33036cf9edb7040f6ee86eb7b032b7776190e95dd3c657b2d98c9ff4a0"
Oct 30 20:23:32.013945 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:23:32.013845    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:23:42.025210 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:23:42.025168    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"]
Oct 30 20:23:42.028092 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:23:42.028067    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5g8vl"]
Oct 30 20:23:42.277554 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:23:42.277476    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c2f926-11c7-43d5-bd4b-e0319700e5f6" path="/var/lib/kubelet/pods/c1c2f926-11c7-43d5-bd4b-e0319700e5f6/volumes"
Oct 30 20:23:53.025701 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:23:53.025662    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"]
Oct 30 20:23:53.029470 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:23:53.029438    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zhvjg"]
Oct 30 20:23:54.277631 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:23:54.277595    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9967e77-6709-4ea7-8908-fd430dac7e6f" path="/var/lib/kubelet/pods/d9967e77-6709-4ea7-8908-fd430dac7e6f/volumes"
Oct 30 20:24:02.013999 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:24:02.013959    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:24:13.022791 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:24:13.022751    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"]
Oct 30 20:24:13.025833 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:24:13.025808    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mc78q"]
Oct 30 20:24:14.277296 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:24:14.277260    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a22111-bc74-4b1b-962f-8940f3dedd1e" path="/var/lib/kubelet/pods/d0a22111-bc74-4b1b-962f-8940f3dedd1e/volumes"
Oct 30 20:24:32.013114 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:24:32.013074    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:24:32.026621 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:24:32.026592    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"]
Oct 30 20:24:32.029982 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:24:32.029955    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2sx5c"]
Oct 30 20:24:32.277788 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:24:32.277698    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="340190c8-9e15-4c76-b559-c84d10ef02d1" path="/var/lib/kubelet/pods/340190c8-9e15-4c76-b559-c84d10ef02d1/volumes"
Oct 30 20:24:51.025582 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:24:51.025469    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"]
Oct 30 20:24:51.027992 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:24:51.027968    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qr6wl"]
Oct 30 20:24:52.278162 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:24:52.278107    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceb0f5c7-f4ef-4e03-a500-64923d854fe9" path="/var/lib/kubelet/pods/ceb0f5c7-f4ef-4e03-a500-64923d854fe9/volumes"
Oct 30 20:25:01.760337 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:25:01.760311    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:25:01.767577 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:25:01.767552    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:25:02.014286 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:25:02.014203    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:25:04.378228 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:25:04.378198    4823 scope.go:122] "RemoveContainer" containerID="824cc3b5517b16d7eaf42e38fba23c468564ff4526d7175816d660b14f45aed6"
Oct 30 20:25:04.387307 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:25:04.387287    4823 scope.go:122] "RemoveContainer" containerID="1b4c1a93a51bba28fc3eaf66e42170cc5a365a6f917f588c1e40f3368ccf10e7"
Oct 30 20:25:08.035140 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:25:08.035103    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"]
Oct 30 20:25:08.039178 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:25:08.039155    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mvxkj"]
Oct 30 20:25:08.278004 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:25:08.277967    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cdc026a-a852-442c-9ce5-b0affb671d30" path="/var/lib/kubelet/pods/9cdc026a-a852-442c-9ce5-b0affb671d30/volumes"
Oct 30 20:25:32.014145 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:25:32.014107    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:25:41.027012 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:25:41.026971    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"]
Oct 30 20:25:41.030762 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:25:41.030734    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-59twx"]
Oct 30 20:25:42.278021 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:25:42.277987    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecefb34c-9b47-4d8a-a5f6-22c5908ac135" path="/var/lib/kubelet/pods/ecefb34c-9b47-4d8a-a5f6-22c5908ac135/volumes"
Oct 30 20:26:02.015396 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:26:02.015356    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:26:04.412650 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:04.412615    4823 scope.go:122] "RemoveContainer" containerID="a9e3c4111d72745d6f7ac7840b76dd0a46ffec26be6baf9f54a72f36f4c1f370"
Oct 30 20:26:04.421978 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:04.421952    4823 scope.go:122] "RemoveContainer" containerID="381c86f5de906154c18bc83773e922c6e79131f9412a865e36e6e63ab6f19592"
Oct 30 20:26:04.434026 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:04.432657    4823 scope.go:122] "RemoveContainer" containerID="497bd4dec29bcab95d71347f6bb57a372e9dcc7880f9fba8e3aa0667aaa0e6ed"
Oct 30 20:26:10.027023 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:10.026986    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"]
Oct 30 20:26:10.030622 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:10.030592    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n2x2f"]
Oct 30 20:26:10.278051 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:10.277945    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcff8490-90ba-4219-86e5-042c9b03de29" path="/var/lib/kubelet/pods/dcff8490-90ba-4219-86e5-042c9b03de29/volumes"
Oct 30 20:26:20.025810 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:20.025769    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"]
Oct 30 20:26:20.029268 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:20.029241    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-8b66z"]
Oct 30 20:26:20.278718 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:20.278632    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158e8d97-91c4-4196-b366-16ee81961f6d" path="/var/lib/kubelet/pods/158e8d97-91c4-4196-b366-16ee81961f6d/volumes"
Oct 30 20:26:32.015610 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:26:32.015565    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:26:35.761299 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.761260    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"]
Oct 30 20:26:35.761907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.761802    4823 cpu_manager.go:414] "containerMap: removing container" podUID="158e8d97-91c4-4196-b366-16ee81961f6d" containerName="revoke-keys"
Oct 30 20:26:35.761907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.761821    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158e8d97-91c4-4196-b366-16ee81961f6d" containerName="revoke-keys"
Oct 30 20:26:35.761996 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.761988    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="158e8d97-91c4-4196-b366-16ee81961f6d" containerName="revoke-keys"
Oct 30 20:26:35.765434 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.765415    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"
Oct 30 20:26:35.768915 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.768888    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:26:35.769170 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.769154    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:26:35.782599 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.782570    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:26:35.782767 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.782643    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:26:35.796690 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.796656    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"]
Oct 30 20:26:35.856795 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.856760    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wrftv\" (UID: \"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"
Oct 30 20:26:35.856990 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.856932    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vln5\" (UniqueName: \"kubernetes.io/projected/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-kube-api-access-5vln5\") pod \"maas-api-revoke-sub-3d4156f1-wrftv\" (UID: \"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"
Oct 30 20:26:35.856990 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.856978    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wrftv\" (UID: \"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"
Oct 30 20:26:35.958300 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.958261    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wrftv\" (UID: \"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"
Oct 30 20:26:35.958497 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.958364    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vln5\" (UniqueName: \"kubernetes.io/projected/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-kube-api-access-5vln5\") pod \"maas-api-revoke-sub-3d4156f1-wrftv\" (UID: \"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"
Oct 30 20:26:35.958497 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.958394    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wrftv\" (UID: \"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"
Oct 30 20:26:35.958944 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.958925    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wrftv\" (UID: \"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"
Oct 30 20:26:35.960821 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.960798    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wrftv\" (UID: \"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"
Oct 30 20:26:35.970550 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:35.970526    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vln5\" (UniqueName: \"kubernetes.io/projected/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-kube-api-access-5vln5\") pod \"maas-api-revoke-sub-3d4156f1-wrftv\" (UID: \"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"
Oct 30 20:26:36.079431 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:36.079342    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"
Oct 30 20:26:36.369294 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:36.369264    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"]
Oct 30 20:26:36.371681 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:26:36.371655    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44fa762_1df6_4c9f_bf03_9bedd1f52fc3.slice/crio-9f86ae2a8249cbbc0616ff85c4facfd6dd77fe1db7fb0b768cca9eb208be232c WatchSource:0}: Error finding container 9f86ae2a8249cbbc0616ff85c4facfd6dd77fe1db7fb0b768cca9eb208be232c: Status 404 returned error can't find the container with id 9f86ae2a8249cbbc0616ff85c4facfd6dd77fe1db7fb0b768cca9eb208be232c
Oct 30 20:26:37.023294 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:37.023207    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv" event={"ID":"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3","Type":"ContainerStarted","Data":"de092405d0157b72456e067796879e7d3249af8ad507f34a0e5d336caef62249"}
Oct 30 20:26:37.023294 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:37.023267    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv" event={"ID":"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3","Type":"ContainerStarted","Data":"9f86ae2a8249cbbc0616ff85c4facfd6dd77fe1db7fb0b768cca9eb208be232c"}
Oct 30 20:26:37.043509 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:37.043451    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv" podStartSLOduration=2.043435512 podStartE2EDuration="2.043435512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:37.040801376 +0000 UTC m=+3396.933415699" watchObservedRunningTime="2026-10-30 20:26:37.043435512 +0000 UTC m=+3396.936049836"
Oct 30 20:26:38.031759 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:38.031717    4823 generic.go:356] "Generic (PLEG): container finished" podID="f44fa762-1df6-4c9f-bf03-9bedd1f52fc3" containerID="de092405d0157b72456e067796879e7d3249af8ad507f34a0e5d336caef62249" exitCode=0
Oct 30 20:26:38.032313 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:38.031755    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv" event={"ID":"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3","Type":"ContainerDied","Data":"de092405d0157b72456e067796879e7d3249af8ad507f34a0e5d336caef62249"}
Oct 30 20:26:39.185071 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:39.185042    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"
Oct 30 20:26:39.291167 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:39.291064    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-maas-api-xks-ca\") pod \"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3\" (UID: \"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3\") "
Oct 30 20:26:39.291363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:39.291210    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-maas-api-openshift-ca\") pod \"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3\" (UID: \"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3\") "
Oct 30 20:26:39.291363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:39.291250    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-kube-api-access-5vln5\" (UniqueName: \"kubernetes.io/projected/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-kube-api-access-5vln5\") pod \"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3\" (UID: \"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3\") "
Oct 30 20:26:39.291612 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:39.291588    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-maas-api-openshift-ca" pod "f44fa762-1df6-4c9f-bf03-9bedd1f52fc3" (UID: "f44fa762-1df6-4c9f-bf03-9bedd1f52fc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:39.293417 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:39.293382    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-maas-api-xks-ca" pod "f44fa762-1df6-4c9f-bf03-9bedd1f52fc3" (UID: "f44fa762-1df6-4c9f-bf03-9bedd1f52fc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:39.293528 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:39.293392    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-kube-api-access-5vln5" pod "f44fa762-1df6-4c9f-bf03-9bedd1f52fc3" (UID: "f44fa762-1df6-4c9f-bf03-9bedd1f52fc3"). InnerVolumeSpecName "kube-api-access-5vln5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:39.392341 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:39.392293    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:39.392341 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:39.392334    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vln5\" (UniqueName: \"kubernetes.io/projected/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-kube-api-access-5vln5\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:39.392341 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:39.392346    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:40.045193 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:40.045152    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv" event={"ID":"f44fa762-1df6-4c9f-bf03-9bedd1f52fc3","Type":"ContainerDied","Data":"9f86ae2a8249cbbc0616ff85c4facfd6dd77fe1db7fb0b768cca9eb208be232c"}
Oct 30 20:26:40.045400 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:40.045207    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f86ae2a8249cbbc0616ff85c4facfd6dd77fe1db7fb0b768cca9eb208be232c"
Oct 30 20:26:40.045400 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:40.045169    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"
Oct 30 20:26:47.309241 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:47.309194    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:26:48.519226 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:48.484963    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"]
Oct 30 20:26:48.519886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:48.519845    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"]
Oct 30 20:26:48.541651 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:48.540465    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9" podUID="8574d565-f870-4374-97ca-7e5bee83f2a3" containerName="istio-proxy" containerID="cri-o://6a71ab5be62dc1dea92cb9ce19137c8d599477f8030d69a6a6ee544cc96ca451" gracePeriod=30
Oct 30 20:26:48.541651 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:48.526118    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx" podUID="88deb5dd-bf6b-406b-9aa9-103319a550e6" containerName="istio-proxy" containerID="cri-o://bb00618104779c69ce0b4df90def3dbf24e91dc460af633bcd25b6686d3dba13" gracePeriod=30
Oct 30 20:26:50.537215 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:50.537171    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-9djkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:26:50.537900 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:50.537828    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx" podUID="88deb5dd-bf6b-406b-9aa9-103319a550e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:26:51.783292 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:51.783254    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:26:54.522623 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.522582    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"]
Oct 30 20:26:54.523142 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.523124    4823 cpu_manager.go:414] "containerMap: removing container" podUID="f44fa762-1df6-4c9f-bf03-9bedd1f52fc3" containerName="revoke-keys"
Oct 30 20:26:54.523142 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.523142    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f44fa762-1df6-4c9f-bf03-9bedd1f52fc3" containerName="revoke-keys"
Oct 30 20:26:54.523292 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.523227    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="f44fa762-1df6-4c9f-bf03-9bedd1f52fc3" containerName="revoke-keys"
Oct 30 20:26:54.526465 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.526442    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.556509 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.556420    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-69ac38-openshift-default-dockercfg-5v7jk\""
Oct 30 20:26:54.565155 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.565116    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"]
Oct 30 20:26:54.597138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.595953    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-credential-socket\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.597138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.596025    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istiod-ca-cert\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.597138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.596087    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-workload-certs\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.597138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.596117    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-podinfo\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.597138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.596156    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-token\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.597138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.596182    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-envoy\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.597138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.596255    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pz2b\" (UniqueName: \"kubernetes.io/projected/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-kube-api-access-8pz2b\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.597138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.596321    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-data\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.597138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.596355    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-workload-socket\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.698931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.698536    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istiod-ca-cert\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.698931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.698626    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-workload-certs\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.698931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.698656    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-podinfo\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.698931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.698742    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-token\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.698931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.698773    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-envoy\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.698931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.698850    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pz2b\" (UniqueName: \"kubernetes.io/projected/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-kube-api-access-8pz2b\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.698931 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.698931    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-data\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.699523 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.698964    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-workload-socket\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.699523 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.699016    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-credential-socket\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.700924 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.700787    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istiod-ca-cert\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.702107 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.701360    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-credential-socket\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.702997 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.702676    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-workload-certs\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.705947 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.703442    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-workload-socket\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.707180 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.707147    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-data\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.713724 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.713647    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-envoy\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.715803 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.715756    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-podinfo\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.716492 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.716450    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-token\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.717210 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.717184    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pz2b\" (UniqueName: \"kubernetes.io/projected/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-kube-api-access-8pz2b\") pod \"e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") " pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:54.769877 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.769783    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 20:26:54.799714 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.799683    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-token\" (UniqueName: \"kubernetes.io/projected/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-token\") pod \"88deb5dd-bf6b-406b-9aa9-103319a550e6\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") "
Oct 30 20:26:54.800082 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.799739    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88deb5dd-bf6b-406b-9aa9-103319a550e6-kube-api-access-zsspr\" (UniqueName: \"kubernetes.io/projected/88deb5dd-bf6b-406b-9aa9-103319a550e6-kube-api-access-zsspr\") pod \"88deb5dd-bf6b-406b-9aa9-103319a550e6\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") "
Oct 30 20:26:54.800082 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.799770    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-workload-socket\") pod \"88deb5dd-bf6b-406b-9aa9-103319a550e6\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") "
Oct 30 20:26:54.800082 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.799815    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-data\") pod \"88deb5dd-bf6b-406b-9aa9-103319a550e6\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") "
Oct 30 20:26:54.800082 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.799841    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-podinfo\") pod \"88deb5dd-bf6b-406b-9aa9-103319a550e6\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") "
Oct 30 20:26:54.800441 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.800415    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-data" pod "88deb5dd-bf6b-406b-9aa9-103319a550e6" (UID: "88deb5dd-bf6b-406b-9aa9-103319a550e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.800511 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.800417    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88deb5dd-bf6b-406b-9aa9-103319a550e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88deb5dd-bf6b-406b-9aa9-103319a550e6-istiod-ca-cert\") pod \"88deb5dd-bf6b-406b-9aa9-103319a550e6\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") "
Oct 30 20:26:54.800511 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.800499    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-envoy\") pod \"88deb5dd-bf6b-406b-9aa9-103319a550e6\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") "
Oct 30 20:26:54.800625 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.800594    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-credential-socket\") pod \"88deb5dd-bf6b-406b-9aa9-103319a550e6\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") "
Oct 30 20:26:54.800692 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.800628    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-workload-certs\") pod \"88deb5dd-bf6b-406b-9aa9-103319a550e6\" (UID: \"88deb5dd-bf6b-406b-9aa9-103319a550e6\") "
Oct 30 20:26:54.800692 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.800683    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88deb5dd-bf6b-406b-9aa9-103319a550e6-istiod-ca-cert" pod "88deb5dd-bf6b-406b-9aa9-103319a550e6" (UID: "88deb5dd-bf6b-406b-9aa9-103319a550e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:54.801116 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.800964    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-workload-socket" pod "88deb5dd-bf6b-406b-9aa9-103319a550e6" (UID: "88deb5dd-bf6b-406b-9aa9-103319a550e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.801216 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.801149    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-workload-certs" pod "88deb5dd-bf6b-406b-9aa9-103319a550e6" (UID: "88deb5dd-bf6b-406b-9aa9-103319a550e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.801367 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.801340    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88deb5dd-bf6b-406b-9aa9-103319a550e6-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.801487 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.801372    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.801487 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.801389    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.801616 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.801594    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-credential-socket" pod "88deb5dd-bf6b-406b-9aa9-103319a550e6" (UID: "88deb5dd-bf6b-406b-9aa9-103319a550e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.806302 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.806257    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-token" pod "88deb5dd-bf6b-406b-9aa9-103319a550e6" (UID: "88deb5dd-bf6b-406b-9aa9-103319a550e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:54.808354 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.808304    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88deb5dd-bf6b-406b-9aa9-103319a550e6-kube-api-access-zsspr" pod "88deb5dd-bf6b-406b-9aa9-103319a550e6" (UID: "88deb5dd-bf6b-406b-9aa9-103319a550e6"). InnerVolumeSpecName "kube-api-access-zsspr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:54.808660 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.808634    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-podinfo" pod "88deb5dd-bf6b-406b-9aa9-103319a550e6" (UID: "88deb5dd-bf6b-406b-9aa9-103319a550e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:26:54.810467 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.810439    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-envoy" pod "88deb5dd-bf6b-406b-9aa9-103319a550e6" (UID: "88deb5dd-bf6b-406b-9aa9-103319a550e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.843702 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.843669    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:26:54.902684 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.902646    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-token\" (UniqueName: \"kubernetes.io/projected/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-token\") pod \"8574d565-f870-4374-97ca-7e5bee83f2a3\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") "
Oct 30 20:26:54.902908 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.902691    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-envoy\") pod \"8574d565-f870-4374-97ca-7e5bee83f2a3\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") "
Oct 30 20:26:54.902908 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.902734    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8574d565-f870-4374-97ca-7e5bee83f2a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8574d565-f870-4374-97ca-7e5bee83f2a3-istiod-ca-cert\") pod \"8574d565-f870-4374-97ca-7e5bee83f2a3\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") "
Oct 30 20:26:54.902908 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.902763    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-credential-socket\") pod \"8574d565-f870-4374-97ca-7e5bee83f2a3\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") "
Oct 30 20:26:54.902908 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.902799    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8574d565-f870-4374-97ca-7e5bee83f2a3-kube-api-access-sqlwk\" (UniqueName: \"kubernetes.io/projected/8574d565-f870-4374-97ca-7e5bee83f2a3-kube-api-access-sqlwk\") pod \"8574d565-f870-4374-97ca-7e5bee83f2a3\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") "
Oct 30 20:26:54.902908 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.902849    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-workload-socket\") pod \"8574d565-f870-4374-97ca-7e5bee83f2a3\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") "
Oct 30 20:26:54.903211 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.902917    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-podinfo\") pod \"8574d565-f870-4374-97ca-7e5bee83f2a3\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") "
Oct 30 20:26:54.903211 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.902955    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-workload-certs\") pod \"8574d565-f870-4374-97ca-7e5bee83f2a3\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") "
Oct 30 20:26:54.903211 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.902996    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-data\") pod \"8574d565-f870-4374-97ca-7e5bee83f2a3\" (UID: \"8574d565-f870-4374-97ca-7e5bee83f2a3\") "
Oct 30 20:26:54.903211 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.903097    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-credential-socket" pod "8574d565-f870-4374-97ca-7e5bee83f2a3" (UID: "8574d565-f870-4374-97ca-7e5bee83f2a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.903211 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.903177    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8574d565-f870-4374-97ca-7e5bee83f2a3-istiod-ca-cert" pod "8574d565-f870-4374-97ca-7e5bee83f2a3" (UID: "8574d565-f870-4374-97ca-7e5bee83f2a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:54.903474 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.903298    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-workload-certs" pod "8574d565-f870-4374-97ca-7e5bee83f2a3" (UID: "8574d565-f870-4374-97ca-7e5bee83f2a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.903474 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.903317    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-data" pod "8574d565-f870-4374-97ca-7e5bee83f2a3" (UID: "8574d565-f870-4374-97ca-7e5bee83f2a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.903474 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.903330    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-workload-socket" pod "8574d565-f870-4374-97ca-7e5bee83f2a3" (UID: "8574d565-f870-4374-97ca-7e5bee83f2a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.903624 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.903557    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.903624 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.903579    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.903624 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.903595    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.903624 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.903610    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsspr\" (UniqueName: \"kubernetes.io/projected/88deb5dd-bf6b-406b-9aa9-103319a550e6-kube-api-access-zsspr\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.903827 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.903624    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.903827 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.903638    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.903827 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.903651    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.903827 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.903667    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.903827 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.903681    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.903827 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.903697    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88deb5dd-bf6b-406b-9aa9-103319a550e6-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.903827 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.903711    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8574d565-f870-4374-97ca-7e5bee83f2a3-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.905293 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.905263    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8574d565-f870-4374-97ca-7e5bee83f2a3-kube-api-access-sqlwk" pod "8574d565-f870-4374-97ca-7e5bee83f2a3" (UID: "8574d565-f870-4374-97ca-7e5bee83f2a3"). InnerVolumeSpecName "kube-api-access-sqlwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:54.905490 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.905470    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-envoy" pod "8574d565-f870-4374-97ca-7e5bee83f2a3" (UID: "8574d565-f870-4374-97ca-7e5bee83f2a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.905584 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.905497    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-token" pod "8574d565-f870-4374-97ca-7e5bee83f2a3" (UID: "8574d565-f870-4374-97ca-7e5bee83f2a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:54.905584 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.905572    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-podinfo" pod "8574d565-f870-4374-97ca-7e5bee83f2a3" (UID: "8574d565-f870-4374-97ca-7e5bee83f2a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:26:54.960604 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:54.960561    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:55.004435 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.004398    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.004435 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.004432    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.004435 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.004442    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8574d565-f870-4374-97ca-7e5bee83f2a3-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.004683 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.004452    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqlwk\" (UniqueName: \"kubernetes.io/projected/8574d565-f870-4374-97ca-7e5bee83f2a3-kube-api-access-sqlwk\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:55.157807 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.157764    4823 generic.go:356] "Generic (PLEG): container finished" podID="88deb5dd-bf6b-406b-9aa9-103319a550e6" containerID="bb00618104779c69ce0b4df90def3dbf24e91dc460af633bcd25b6686d3dba13" exitCode=0
Oct 30 20:26:55.158049 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.157836    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"
Oct 30 20:26:55.158049 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.157851    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx" event={"ID":"88deb5dd-bf6b-406b-9aa9-103319a550e6","Type":"ContainerDied","Data":"bb00618104779c69ce0b4df90def3dbf24e91dc460af633bcd25b6686d3dba13"}
Oct 30 20:26:55.158049 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.157955    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx" event={"ID":"88deb5dd-bf6b-406b-9aa9-103319a550e6","Type":"ContainerDied","Data":"080d67cc2f8336682386c3984dd4e4f8c184ffaf7dfead32f8c5f38de2a126b9"}
Oct 30 20:26:55.158049 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.157973    4823 scope.go:122] "RemoveContainer" containerID="bb00618104779c69ce0b4df90def3dbf24e91dc460af633bcd25b6686d3dba13"
Oct 30 20:26:55.159532 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.159504    4823 generic.go:356] "Generic (PLEG): container finished" podID="8574d565-f870-4374-97ca-7e5bee83f2a3" containerID="6a71ab5be62dc1dea92cb9ce19137c8d599477f8030d69a6a6ee544cc96ca451" exitCode=0
Oct 30 20:26:55.159657 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.159579    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"
Oct 30 20:26:55.159657 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.159583    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9" event={"ID":"8574d565-f870-4374-97ca-7e5bee83f2a3","Type":"ContainerDied","Data":"6a71ab5be62dc1dea92cb9ce19137c8d599477f8030d69a6a6ee544cc96ca451"}
Oct 30 20:26:55.159657 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.159631    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9" event={"ID":"8574d565-f870-4374-97ca-7e5bee83f2a3","Type":"ContainerDied","Data":"ef323fbcbc9b6ce3c561343baa238671a671278131f99f1229d32af4c65d31a2"}
Oct 30 20:26:55.171378 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.171348    4823 scope.go:122] "RemoveContainer" containerID="bb00618104779c69ce0b4df90def3dbf24e91dc460af633bcd25b6686d3dba13"
Oct 30 20:26:55.171710 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:26:55.171690    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb00618104779c69ce0b4df90def3dbf24e91dc460af633bcd25b6686d3dba13\": container with ID starting with bb00618104779c69ce0b4df90def3dbf24e91dc460af633bcd25b6686d3dba13 not found: ID does not exist" containerID="bb00618104779c69ce0b4df90def3dbf24e91dc460af633bcd25b6686d3dba13"
Oct 30 20:26:55.171792 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.171721    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb00618104779c69ce0b4df90def3dbf24e91dc460af633bcd25b6686d3dba13"} err="failed to get container status \"bb00618104779c69ce0b4df90def3dbf24e91dc460af633bcd25b6686d3dba13\": rpc error: code = NotFound desc = could not find container \"bb00618104779c69ce0b4df90def3dbf24e91dc460af633bcd25b6686d3dba13\": container with ID starting with bb00618104779c69ce0b4df90def3dbf24e91dc460af633bcd25b6686d3dba13 not found: ID does not exist"
Oct 30 20:26:55.171792 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.171739    4823 scope.go:122] "RemoveContainer" containerID="6a71ab5be62dc1dea92cb9ce19137c8d599477f8030d69a6a6ee544cc96ca451"
Oct 30 20:26:55.181011 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.180993    4823 scope.go:122] "RemoveContainer" containerID="6a71ab5be62dc1dea92cb9ce19137c8d599477f8030d69a6a6ee544cc96ca451"
Oct 30 20:26:55.181311 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:26:55.181290    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a71ab5be62dc1dea92cb9ce19137c8d599477f8030d69a6a6ee544cc96ca451\": container with ID starting with 6a71ab5be62dc1dea92cb9ce19137c8d599477f8030d69a6a6ee544cc96ca451 not found: ID does not exist" containerID="6a71ab5be62dc1dea92cb9ce19137c8d599477f8030d69a6a6ee544cc96ca451"
Oct 30 20:26:55.181399 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.181318    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a71ab5be62dc1dea92cb9ce19137c8d599477f8030d69a6a6ee544cc96ca451"} err="failed to get container status \"6a71ab5be62dc1dea92cb9ce19137c8d599477f8030d69a6a6ee544cc96ca451\": rpc error: code = NotFound desc = could not find container \"6a71ab5be62dc1dea92cb9ce19137c8d599477f8030d69a6a6ee544cc96ca451\": container with ID starting with 6a71ab5be62dc1dea92cb9ce19137c8d599477f8030d69a6a6ee544cc96ca451 not found: ID does not exist"
Oct 30 20:26:55.231588 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.231555    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"]
Oct 30 20:26:55.255606 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.255563    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8krz9"]
Oct 30 20:26:55.303531 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.303497    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"]
Oct 30 20:26:55.311214 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.310029    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"]
Oct 30 20:26:55.312165 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.312138    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-9djkx"]
Oct 30 20:26:55.313080 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:26:55.313051    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3d8e2b_c5fb_4d10_8ff1_c69f4592df79.slice/crio-3a64466f24ecbf2b526f50c9ae0cb44d9d3629ba054ff02e4745e1ee2a5d7b83 WatchSource:0}: Error finding container 3a64466f24ecbf2b526f50c9ae0cb44d9d3629ba054ff02e4745e1ee2a5d7b83: Status 404 returned error can't find the container with id 3a64466f24ecbf2b526f50c9ae0cb44d9d3629ba054ff02e4745e1ee2a5d7b83
Oct 30 20:26:55.315332 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.315301    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:26:55.315450 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.315379    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:26:55.315450 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:55.315424    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:26:56.167183 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:56.167143    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm" event={"ID":"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79","Type":"ContainerStarted","Data":"f6538f0e26664e78d86311c63cf11ce770c309ceeb53495fc9161f0e149fdf74"}
Oct 30 20:26:56.167691 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:56.167213    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm" event={"ID":"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79","Type":"ContainerStarted","Data":"3a64466f24ecbf2b526f50c9ae0cb44d9d3629ba054ff02e4745e1ee2a5d7b83"}
Oct 30 20:26:56.192893 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:56.192807    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm" podStartSLOduration=2.192790748 podStartE2EDuration="2.192790748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:56.192316852 +0000 UTC m=+3416.084931176" watchObservedRunningTime="2026-10-30 20:26:56.192790748 +0000 UTC m=+3416.085405072"
Oct 30 20:26:56.277886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:56.277829    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8574d565-f870-4374-97ca-7e5bee83f2a3" path="/var/lib/kubelet/pods/8574d565-f870-4374-97ca-7e5bee83f2a3/volumes"
Oct 30 20:26:56.278355 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:56.278339    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88deb5dd-bf6b-406b-9aa9-103319a550e6" path="/var/lib/kubelet/pods/88deb5dd-bf6b-406b-9aa9-103319a550e6/volumes"
Oct 30 20:26:56.961647 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:56.961602    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:56.967185 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:56.967155    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:57.174885 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:57.174828    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:26:57.175985 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:26:57.175967    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:27:02.016038 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:27:02.015996    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:27:02.394486 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:02.394444    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:27:04.466621 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:04.466586    4823 scope.go:122] "RemoveContainer" containerID="26ba9e4b3fbf2ca08127de96e032081a7746ca40b9c3b6f7ec4e9c5f3c6912ba"
Oct 30 20:27:04.477435 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:04.477409    4823 scope.go:122] "RemoveContainer" containerID="50c1cfaf1e69c38f60f13c7dbb89a5589f8e458968f6fcd0e578a05331f69f71"
Oct 30 20:27:04.486570 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:04.486545    4823 scope.go:122] "RemoveContainer" containerID="f93484268ef08a285fa046f4ae4c863d289d181a315d4843a388ca4b8fd15ace"
Oct 30 20:27:15.468507 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.468470    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"]
Oct 30 20:27:15.469039 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.468947    4823 cpu_manager.go:414] "containerMap: removing container" podUID="88deb5dd-bf6b-406b-9aa9-103319a550e6" containerName="istio-proxy"
Oct 30 20:27:15.469039 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.468960    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88deb5dd-bf6b-406b-9aa9-103319a550e6" containerName="istio-proxy"
Oct 30 20:27:15.469039 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.469008    4823 cpu_manager.go:414] "containerMap: removing container" podUID="8574d565-f870-4374-97ca-7e5bee83f2a3" containerName="istio-proxy"
Oct 30 20:27:15.469039 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.469016    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8574d565-f870-4374-97ca-7e5bee83f2a3" containerName="istio-proxy"
Oct 30 20:27:15.469209 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.469148    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="8574d565-f870-4374-97ca-7e5bee83f2a3" containerName="istio-proxy"
Oct 30 20:27:15.469209 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.469167    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="88deb5dd-bf6b-406b-9aa9-103319a550e6" containerName="istio-proxy"
Oct 30 20:27:15.474625 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.474599    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.480588 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.480467    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-69ac38\"/\"openshift-service-ca.crt\""
Oct 30 20:27:15.482477 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.482434    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-69ac38\"/\"kube-root-ca.crt\""
Oct 30 20:27:15.482798 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.482770    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-69ac38\"/\"simulator-single-main-69ac38-kserve-self-signed-certs\""
Oct 30 20:27:15.483070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.483050    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-69ac38\"/\"default-dockercfg-mcdns\""
Oct 30 20:27:15.486918 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.486734    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"]
Oct 30 20:27:15.526836 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.526794    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-dshm\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.527067 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.526846    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwx28\" (UniqueName: \"kubernetes.io/projected/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-kube-api-access-fwx28\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.527067 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.526995    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-tls-certs\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.527067 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.527041    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-tmp-dir\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.527273 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.527160    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-home\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.527273 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.527200    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-model-cache\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.628064 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.628010    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-dshm\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.628064 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.628061    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwx28\" (UniqueName: \"kubernetes.io/projected/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-kube-api-access-fwx28\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.628364 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.628103    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-tls-certs\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.628364 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.628131    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-tmp-dir\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.628364 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.628207    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-home\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.628364 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.628258    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-model-cache\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.628616 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.628592    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-tmp-dir\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.628729 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.628705    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-model-cache\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.628812 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.628789    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-home\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.630592 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.630570    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-dshm\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.631066 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.631044    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-tls-certs\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.640789 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.640578    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwx28\" (UniqueName: \"kubernetes.io/projected/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-kube-api-access-fwx28\") pod \"simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") " pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:15.794574 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:15.794480    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:16.090580 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:16.090537    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"]
Oct 30 20:27:16.093549 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:27:16.093514    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod828c3ad9_e0a7_479c_bbc6_7da1820a5ac0.slice/crio-8c2f24835f469b21ca546163c5a866d5b6c6dad6448d13c766c9b1a22815fef8 WatchSource:0}: Error finding container 8c2f24835f469b21ca546163c5a866d5b6c6dad6448d13c766c9b1a22815fef8: Status 404 returned error can't find the container with id 8c2f24835f469b21ca546163c5a866d5b6c6dad6448d13c766c9b1a22815fef8
Oct 30 20:27:16.306046 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:16.306000    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6" event={"ID":"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0","Type":"ContainerStarted","Data":"54b37523a5ab1e35c0010324eff11ccf17c1dea5e0347a9d92146bb36930283e"}
Oct 30 20:27:16.306240 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:16.306058    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6" event={"ID":"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0","Type":"ContainerStarted","Data":"8c2f24835f469b21ca546163c5a866d5b6c6dad6448d13c766c9b1a22815fef8"}
Oct 30 20:27:16.325982 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:16.325925    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6" podStartSLOduration=1.325908747 podStartE2EDuration="1.325908747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:16.323927038 +0000 UTC m=+3436.216541361" watchObservedRunningTime="2026-10-30 20:27:16.325908747 +0000 UTC m=+3436.218523065"
Oct 30 20:27:16.795508 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:16.795468    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:17.255359 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:17.255319    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:27:18.570976 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.570891    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"]
Oct 30 20:27:18.574547 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.574522    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.604500 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.604459    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"]
Oct 30 20:27:18.661259 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.661149    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c81047-57b2-4136-a7c7-cd930e059677-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.661259 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.661204    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c81047-57b2-4136-a7c7-cd930e059677-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.661479 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.661300    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jsrm\" (UniqueName: \"kubernetes.io/projected/f3c81047-57b2-4136-a7c7-cd930e059677-kube-api-access-9jsrm\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.661479 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.661361    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c81047-57b2-4136-a7c7-cd930e059677-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.661479 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.661393    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3c81047-57b2-4136-a7c7-cd930e059677-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.661592 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.661475    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c81047-57b2-4136-a7c7-cd930e059677-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.661592 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.661523    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c81047-57b2-4136-a7c7-cd930e059677-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.661592 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.661557    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c81047-57b2-4136-a7c7-cd930e059677-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.661691 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.661633    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c81047-57b2-4136-a7c7-cd930e059677-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.763025 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.762983    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c81047-57b2-4136-a7c7-cd930e059677-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.763025 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.763031    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c81047-57b2-4136-a7c7-cd930e059677-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.763314 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.763057    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jsrm\" (UniqueName: \"kubernetes.io/projected/f3c81047-57b2-4136-a7c7-cd930e059677-kube-api-access-9jsrm\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.763314 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.763110    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c81047-57b2-4136-a7c7-cd930e059677-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.763314 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.763130    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3c81047-57b2-4136-a7c7-cd930e059677-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.763314 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.763236    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c81047-57b2-4136-a7c7-cd930e059677-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.763314 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.763286    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c81047-57b2-4136-a7c7-cd930e059677-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.763594 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.763338    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c81047-57b2-4136-a7c7-cd930e059677-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.763594 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.763389    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c81047-57b2-4136-a7c7-cd930e059677-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.763718 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.763629    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c81047-57b2-4136-a7c7-cd930e059677-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.763718 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.763658    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c81047-57b2-4136-a7c7-cd930e059677-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.763838 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.763790    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c81047-57b2-4136-a7c7-cd930e059677-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.763935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.763914    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c81047-57b2-4136-a7c7-cd930e059677-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.763982 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.763955    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c81047-57b2-4136-a7c7-cd930e059677-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.765443 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.765424    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c81047-57b2-4136-a7c7-cd930e059677-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.765825 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.765807    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c81047-57b2-4136-a7c7-cd930e059677-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.791470 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.791440    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jsrm\" (UniqueName: \"kubernetes.io/projected/f3c81047-57b2-4136-a7c7-cd930e059677-kube-api-access-9jsrm\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.796848 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.796806    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3c81047-57b2-4136-a7c7-cd930e059677-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rl8qk\" (UID: \"f3c81047-57b2-4136-a7c7-cd930e059677\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:18.888274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:18.888241    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:19.187810 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:19.187781    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"]
Oct 30 20:27:19.189927 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:27:19.189900    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c81047_57b2_4136_a7c7_cd930e059677.slice/crio-272739dafb7d126f137b2b78115a3372936742296d433e32fb93e20e96e9d242 WatchSource:0}: Error finding container 272739dafb7d126f137b2b78115a3372936742296d433e32fb93e20e96e9d242: Status 404 returned error can't find the container with id 272739dafb7d126f137b2b78115a3372936742296d433e32fb93e20e96e9d242
Oct 30 20:27:19.192382 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:19.192327    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:27:19.192469 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:19.192391    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:27:19.192469 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:19.192425    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:27:19.326832 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:19.326787    4823 generic.go:356] "Generic (PLEG): container finished" podID="9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" containerID="f6538f0e26664e78d86311c63cf11ce770c309ceeb53495fc9161f0e149fdf74" exitCode=0
Oct 30 20:27:19.327092 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:19.326853    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm" event={"ID":"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79","Type":"ContainerDied","Data":"f6538f0e26664e78d86311c63cf11ce770c309ceeb53495fc9161f0e149fdf74"}
Oct 30 20:27:19.327352 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:19.327329    4823 scope.go:122] "RemoveContainer" containerID="f6538f0e26664e78d86311c63cf11ce770c309ceeb53495fc9161f0e149fdf74"
Oct 30 20:27:19.328766 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:19.328739    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk" event={"ID":"f3c81047-57b2-4136-a7c7-cd930e059677","Type":"ContainerStarted","Data":"b055886a76fc7c7bcc09820c8a383742da6d7ebd70894f8c3832a113fef0ea13"}
Oct 30 20:27:19.328926 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:19.328792    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk" event={"ID":"f3c81047-57b2-4136-a7c7-cd930e059677","Type":"ContainerStarted","Data":"272739dafb7d126f137b2b78115a3372936742296d433e32fb93e20e96e9d242"}
Oct 30 20:27:19.328926 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:19.328741    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:27:19.328926 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:19.328890    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:27:19.329122 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:19.328933    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:27:19.417084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:19.417023    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk" podStartSLOduration=1.417003524 podStartE2EDuration="1.417003524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:19.415373378 +0000 UTC m=+3439.307987702" watchObservedRunningTime="2026-10-30 20:27:19.417003524 +0000 UTC m=+3439.309617848"
Oct 30 20:27:19.888369 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:19.888337    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:20.336125 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:20.336032    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm" event={"ID":"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79","Type":"ContainerStarted","Data":"325a42d960bf5bf468f229629f701d99875e6df66f587d3c7c443ce43ce21a8a"}
Oct 30 20:27:20.903907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:20.903145    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rl8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 20:27:20.903907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:20.903215    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk" podUID="f3c81047-57b2-4136-a7c7-cd930e059677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 20:27:20.965665 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:20.965631    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:27:20.966056 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:20.966032    4823 patch_prober.go:29] interesting pod/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 20:27:20.966278 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:20.966251    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm" podUID="9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 20:27:21.889720 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:21.889243    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rl8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 20:27:21.889720 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:21.889316    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk" podUID="f3c81047-57b2-4136-a7c7-cd930e059677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 20:27:21.966345 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:21.966301    4823 patch_prober.go:29] interesting pod/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 20:27:21.967390 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:21.966964    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm" podUID="9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 20:27:22.906342 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:22.906302    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rl8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 20:27:22.906626 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:22.906594    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk" podUID="f3c81047-57b2-4136-a7c7-cd930e059677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 20:27:22.975888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:22.968235    4823 patch_prober.go:29] interesting pod/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 20:27:22.975888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:22.968313    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm" podUID="9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 20:27:23.893793 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:23.893755    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rl8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 20:27:23.894128 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:23.894092    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk" podUID="f3c81047-57b2-4136-a7c7-cd930e059677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 20:27:23.969886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:23.962478    4823 patch_prober.go:29] interesting pod/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 20:27:23.969886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:23.962562    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm" podUID="9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 20:27:24.889092 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:24.889054    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rl8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 20:27:24.889751 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:24.889717    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk" podUID="f3c81047-57b2-4136-a7c7-cd930e059677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 20:27:24.963314 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:24.963273    4823 patch_prober.go:29] interesting pod/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 20:27:24.963615 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:24.963579    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm" podUID="9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 20:27:25.794875 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:25.794831    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:25.815321 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:25.815281    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:25.890545 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:25.888903    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rl8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 20:27:25.890545 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:25.888988    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk" podUID="f3c81047-57b2-4136-a7c7-cd930e059677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 20:27:25.966749 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:25.966714    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:27:25.967319 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:25.967245    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:27:25.969086 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:25.969058    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:27:26.438126 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:26.438086    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:27:26.889004 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:26.888968    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rl8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 20:27:26.889223 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:26.889032    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk" podUID="f3c81047-57b2-4136-a7c7-cd930e059677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 20:27:27.893517 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:27.893487    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:27.893964 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:27.893791    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:27.894568 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:27.894546    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rl8qk"
Oct 30 20:27:32.015955 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:27:32.015911    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:27:34.517968 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:34.517926    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:27:45.944292 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:45.944246    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"]
Oct 30 20:27:45.948397 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:45.948370    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"
Oct 30 20:27:45.954841 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:45.954798    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:27:45.955762 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:45.955739    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:27:45.955762 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:45.955753    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:27:45.955995 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:45.955760    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:27:45.962186 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:45.962141    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"]
Oct 30 20:27:45.991248 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:45.991200    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3e6ac7-bf62-4074-8b84-2903a30744c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c4ea9e5-5qbsj\" (UID: \"cb3e6ac7-bf62-4074-8b84-2903a30744c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"
Oct 30 20:27:45.991470 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:45.991274    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wffgv\" (UniqueName: \"kubernetes.io/projected/cb3e6ac7-bf62-4074-8b84-2903a30744c3-kube-api-access-wffgv\") pod \"maas-api-revoke-sub-3c4ea9e5-5qbsj\" (UID: \"cb3e6ac7-bf62-4074-8b84-2903a30744c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"
Oct 30 20:27:45.991470 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:45.991432    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e6ac7-bf62-4074-8b84-2903a30744c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c4ea9e5-5qbsj\" (UID: \"cb3e6ac7-bf62-4074-8b84-2903a30744c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"
Oct 30 20:27:46.092675 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:46.092589    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e6ac7-bf62-4074-8b84-2903a30744c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c4ea9e5-5qbsj\" (UID: \"cb3e6ac7-bf62-4074-8b84-2903a30744c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"
Oct 30 20:27:46.093041 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:46.092758    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3e6ac7-bf62-4074-8b84-2903a30744c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c4ea9e5-5qbsj\" (UID: \"cb3e6ac7-bf62-4074-8b84-2903a30744c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"
Oct 30 20:27:46.093041 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:46.092792    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wffgv\" (UniqueName: \"kubernetes.io/projected/cb3e6ac7-bf62-4074-8b84-2903a30744c3-kube-api-access-wffgv\") pod \"maas-api-revoke-sub-3c4ea9e5-5qbsj\" (UID: \"cb3e6ac7-bf62-4074-8b84-2903a30744c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"
Oct 30 20:27:46.093324 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:46.093302    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e6ac7-bf62-4074-8b84-2903a30744c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c4ea9e5-5qbsj\" (UID: \"cb3e6ac7-bf62-4074-8b84-2903a30744c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"
Oct 30 20:27:46.095212 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:46.095184    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3e6ac7-bf62-4074-8b84-2903a30744c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c4ea9e5-5qbsj\" (UID: \"cb3e6ac7-bf62-4074-8b84-2903a30744c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"
Oct 30 20:27:46.105360 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:46.105333    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wffgv\" (UniqueName: \"kubernetes.io/projected/cb3e6ac7-bf62-4074-8b84-2903a30744c3-kube-api-access-wffgv\") pod \"maas-api-revoke-sub-3c4ea9e5-5qbsj\" (UID: \"cb3e6ac7-bf62-4074-8b84-2903a30744c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"
Oct 30 20:27:46.263447 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:46.263408    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"
Oct 30 20:27:46.555925 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:46.555885    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"]
Oct 30 20:27:46.558702 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:27:46.558672    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3e6ac7_bf62_4074_8b84_2903a30744c3.slice/crio-60697c7599f331a86db9f1165363ca981586f13f91a2367dda77ffcf9b2d4e0f WatchSource:0}: Error finding container 60697c7599f331a86db9f1165363ca981586f13f91a2367dda77ffcf9b2d4e0f: Status 404 returned error can't find the container with id 60697c7599f331a86db9f1165363ca981586f13f91a2367dda77ffcf9b2d4e0f
Oct 30 20:27:47.546618 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:47.546581    4823 generic.go:356] "Generic (PLEG): container finished" podID="cb3e6ac7-bf62-4074-8b84-2903a30744c3" containerID="87bdcabb1d9983a2a60849d3c837854f4901d1239f8c68b9fccad3380ccf8b16" exitCode=0
Oct 30 20:27:47.547038 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:47.546633    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj" event={"ID":"cb3e6ac7-bf62-4074-8b84-2903a30744c3","Type":"ContainerDied","Data":"87bdcabb1d9983a2a60849d3c837854f4901d1239f8c68b9fccad3380ccf8b16"}
Oct 30 20:27:47.547038 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:47.546716    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj" event={"ID":"cb3e6ac7-bf62-4074-8b84-2903a30744c3","Type":"ContainerStarted","Data":"60697c7599f331a86db9f1165363ca981586f13f91a2367dda77ffcf9b2d4e0f"}
Oct 30 20:27:48.703395 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:48.703367    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"
Oct 30 20:27:48.822914 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:48.822789    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3e6ac7-bf62-4074-8b84-2903a30744c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e6ac7-bf62-4074-8b84-2903a30744c3-maas-api-openshift-ca\") pod \"cb3e6ac7-bf62-4074-8b84-2903a30744c3\" (UID: \"cb3e6ac7-bf62-4074-8b84-2903a30744c3\") "
Oct 30 20:27:48.822914 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:48.822836    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb3e6ac7-bf62-4074-8b84-2903a30744c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3e6ac7-bf62-4074-8b84-2903a30744c3-maas-api-xks-ca\") pod \"cb3e6ac7-bf62-4074-8b84-2903a30744c3\" (UID: \"cb3e6ac7-bf62-4074-8b84-2903a30744c3\") "
Oct 30 20:27:48.822914 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:48.822879    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3e6ac7-bf62-4074-8b84-2903a30744c3-kube-api-access-wffgv\" (UniqueName: \"kubernetes.io/projected/cb3e6ac7-bf62-4074-8b84-2903a30744c3-kube-api-access-wffgv\") pod \"cb3e6ac7-bf62-4074-8b84-2903a30744c3\" (UID: \"cb3e6ac7-bf62-4074-8b84-2903a30744c3\") "
Oct 30 20:27:48.823241 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:48.823189    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3e6ac7-bf62-4074-8b84-2903a30744c3-maas-api-openshift-ca" pod "cb3e6ac7-bf62-4074-8b84-2903a30744c3" (UID: "cb3e6ac7-bf62-4074-8b84-2903a30744c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:48.825084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:48.825060    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3e6ac7-bf62-4074-8b84-2903a30744c3-kube-api-access-wffgv" pod "cb3e6ac7-bf62-4074-8b84-2903a30744c3" (UID: "cb3e6ac7-bf62-4074-8b84-2903a30744c3"). InnerVolumeSpecName "kube-api-access-wffgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:48.825231 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:48.825208    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3e6ac7-bf62-4074-8b84-2903a30744c3-maas-api-xks-ca" pod "cb3e6ac7-bf62-4074-8b84-2903a30744c3" (UID: "cb3e6ac7-bf62-4074-8b84-2903a30744c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:27:48.924277 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:48.924240    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e6ac7-bf62-4074-8b84-2903a30744c3-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:27:48.924277 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:48.924275    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3e6ac7-bf62-4074-8b84-2903a30744c3-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:27:48.924277 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:48.924285    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wffgv\" (UniqueName: \"kubernetes.io/projected/cb3e6ac7-bf62-4074-8b84-2903a30744c3-kube-api-access-wffgv\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:27:49.560825 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:49.560789    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"
Oct 30 20:27:49.561034 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:49.560825    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj" event={"ID":"cb3e6ac7-bf62-4074-8b84-2903a30744c3","Type":"ContainerDied","Data":"60697c7599f331a86db9f1165363ca981586f13f91a2367dda77ffcf9b2d4e0f"}
Oct 30 20:27:49.561034 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:49.560885    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60697c7599f331a86db9f1165363ca981586f13f91a2367dda77ffcf9b2d4e0f"
Oct 30 20:27:57.060322 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.060283    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"]
Oct 30 20:27:57.060845 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.060778    4823 cpu_manager.go:414] "containerMap: removing container" podUID="cb3e6ac7-bf62-4074-8b84-2903a30744c3" containerName="revoke-keys"
Oct 30 20:27:57.060845 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.060792    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3e6ac7-bf62-4074-8b84-2903a30744c3" containerName="revoke-keys"
Oct 30 20:27:57.060971 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.060911    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3e6ac7-bf62-4074-8b84-2903a30744c3" containerName="revoke-keys"
Oct 30 20:27:57.064446 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.064423    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"
Oct 30 20:27:57.073457 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.073430    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:27:57.073653 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.073455    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:27:57.073763 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.073522    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:27:57.073882 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.073578    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:27:57.075033 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.074995    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"]
Oct 30 20:27:57.116499 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.116447    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99b6z\" (UniqueName: \"kubernetes.io/projected/e52412a6-bd5d-477e-af3b-12095d7f5529-kube-api-access-99b6z\") pod \"maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n\" (UID: \"e52412a6-bd5d-477e-af3b-12095d7f5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"
Oct 30 20:27:57.116499 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.116497    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52412a6-bd5d-477e-af3b-12095d7f5529-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n\" (UID: \"e52412a6-bd5d-477e-af3b-12095d7f5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"
Oct 30 20:27:57.116732 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.116588    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52412a6-bd5d-477e-af3b-12095d7f5529-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n\" (UID: \"e52412a6-bd5d-477e-af3b-12095d7f5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"
Oct 30 20:27:57.218110 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.218063    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99b6z\" (UniqueName: \"kubernetes.io/projected/e52412a6-bd5d-477e-af3b-12095d7f5529-kube-api-access-99b6z\") pod \"maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n\" (UID: \"e52412a6-bd5d-477e-af3b-12095d7f5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"
Oct 30 20:27:57.218316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.218121    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52412a6-bd5d-477e-af3b-12095d7f5529-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n\" (UID: \"e52412a6-bd5d-477e-af3b-12095d7f5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"
Oct 30 20:27:57.218316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.218162    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52412a6-bd5d-477e-af3b-12095d7f5529-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n\" (UID: \"e52412a6-bd5d-477e-af3b-12095d7f5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"
Oct 30 20:27:57.218803 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.218783    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52412a6-bd5d-477e-af3b-12095d7f5529-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n\" (UID: \"e52412a6-bd5d-477e-af3b-12095d7f5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"
Oct 30 20:27:57.220702 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.220681    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52412a6-bd5d-477e-af3b-12095d7f5529-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n\" (UID: \"e52412a6-bd5d-477e-af3b-12095d7f5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"
Oct 30 20:27:57.240767 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.240729    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99b6z\" (UniqueName: \"kubernetes.io/projected/e52412a6-bd5d-477e-af3b-12095d7f5529-kube-api-access-99b6z\") pod \"maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n\" (UID: \"e52412a6-bd5d-477e-af3b-12095d7f5529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"
Oct 30 20:27:57.379650 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.379611    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"
Oct 30 20:27:57.663076 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:57.663000    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"]
Oct 30 20:27:57.666877 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:27:57.666836    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52412a6_bd5d_477e_af3b_12095d7f5529.slice/crio-89c89cf14f25e16b219ac3ff05414b9ec7e23017f485e5c3589b0111c9ed1e44 WatchSource:0}: Error finding container 89c89cf14f25e16b219ac3ff05414b9ec7e23017f485e5c3589b0111c9ed1e44: Status 404 returned error can't find the container with id 89c89cf14f25e16b219ac3ff05414b9ec7e23017f485e5c3589b0111c9ed1e44
Oct 30 20:27:58.011077 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:58.011031    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:27:58.619421 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:58.619378    4823 generic.go:356] "Generic (PLEG): container finished" podID="e52412a6-bd5d-477e-af3b-12095d7f5529" containerID="e7bf22b00287c47b6b5e2b32430e8e8cca2ef784559787d85c77ca3cd8d6de79" exitCode=0
Oct 30 20:27:58.619830 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:58.619476    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n" event={"ID":"e52412a6-bd5d-477e-af3b-12095d7f5529","Type":"ContainerDied","Data":"e7bf22b00287c47b6b5e2b32430e8e8cca2ef784559787d85c77ca3cd8d6de79"}
Oct 30 20:27:58.619830 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:58.619549    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n" event={"ID":"e52412a6-bd5d-477e-af3b-12095d7f5529","Type":"ContainerStarted","Data":"89c89cf14f25e16b219ac3ff05414b9ec7e23017f485e5c3589b0111c9ed1e44"}
Oct 30 20:27:59.779000 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:59.778971    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"
Oct 30 20:27:59.846981 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:59.846946    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52412a6-bd5d-477e-af3b-12095d7f5529-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52412a6-bd5d-477e-af3b-12095d7f5529-maas-api-openshift-ca\") pod \"e52412a6-bd5d-477e-af3b-12095d7f5529\" (UID: \"e52412a6-bd5d-477e-af3b-12095d7f5529\") "
Oct 30 20:27:59.847188 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:59.847058    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52412a6-bd5d-477e-af3b-12095d7f5529-kube-api-access-99b6z\" (UniqueName: \"kubernetes.io/projected/e52412a6-bd5d-477e-af3b-12095d7f5529-kube-api-access-99b6z\") pod \"e52412a6-bd5d-477e-af3b-12095d7f5529\" (UID: \"e52412a6-bd5d-477e-af3b-12095d7f5529\") "
Oct 30 20:27:59.847188 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:59.847102    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e52412a6-bd5d-477e-af3b-12095d7f5529-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52412a6-bd5d-477e-af3b-12095d7f5529-maas-api-xks-ca\") pod \"e52412a6-bd5d-477e-af3b-12095d7f5529\" (UID: \"e52412a6-bd5d-477e-af3b-12095d7f5529\") "
Oct 30 20:27:59.847387 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:59.847357    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52412a6-bd5d-477e-af3b-12095d7f5529-maas-api-openshift-ca" pod "e52412a6-bd5d-477e-af3b-12095d7f5529" (UID: "e52412a6-bd5d-477e-af3b-12095d7f5529"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:59.849279 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:59.849253    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e52412a6-bd5d-477e-af3b-12095d7f5529-maas-api-xks-ca" pod "e52412a6-bd5d-477e-af3b-12095d7f5529" (UID: "e52412a6-bd5d-477e-af3b-12095d7f5529"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:27:59.849393 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:59.849355    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52412a6-bd5d-477e-af3b-12095d7f5529-kube-api-access-99b6z" pod "e52412a6-bd5d-477e-af3b-12095d7f5529" (UID: "e52412a6-bd5d-477e-af3b-12095d7f5529"). InnerVolumeSpecName "kube-api-access-99b6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:59.948815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:59.948702    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99b6z\" (UniqueName: \"kubernetes.io/projected/e52412a6-bd5d-477e-af3b-12095d7f5529-kube-api-access-99b6z\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:27:59.948815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:59.948741    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52412a6-bd5d-477e-af3b-12095d7f5529-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:27:59.948815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:27:59.948754    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52412a6-bd5d-477e-af3b-12095d7f5529-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:00.632085 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:00.632050    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"
Oct 30 20:28:00.632270 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:00.632082    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n" event={"ID":"e52412a6-bd5d-477e-af3b-12095d7f5529","Type":"ContainerDied","Data":"89c89cf14f25e16b219ac3ff05414b9ec7e23017f485e5c3589b0111c9ed1e44"}
Oct 30 20:28:00.632270 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:00.632145    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89c89cf14f25e16b219ac3ff05414b9ec7e23017f485e5c3589b0111c9ed1e44"
Oct 30 20:28:01.843057 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:01.843017    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"]
Oct 30 20:28:02.014977 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:28:02.014938    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:28:04.532507 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:04.532469    4823 scope.go:122] "RemoveContainer" containerID="be3c8de72fc64acdd14c1c2c233eef80727683869de0e302906474f33639bdc5"
Oct 30 20:28:04.541910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:04.541887    4823 scope.go:122] "RemoveContainer" containerID="29bd4f2ee5197d3c9fe85a00372257bb13ad098a5e50999b197f7feb77b0adc0"
Oct 30 20:28:08.493111 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:08.493066    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"]
Oct 30 20:28:08.501702 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:08.501658    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-69ac38-z6d5n"]
Oct 30 20:28:10.028645 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:10.028608    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:28:10.278120 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:10.278083    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52412a6-bd5d-477e-af3b-12095d7f5529" path="/var/lib/kubelet/pods/e52412a6-bd5d-477e-af3b-12095d7f5529/volumes"
Oct 30 20:28:15.577244 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:15.577205    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"]
Oct 30 20:28:15.577749 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:15.577510    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm" podUID="9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" containerName="istio-proxy" containerID="cri-o://325a42d960bf5bf468f229629f701d99875e6df66f587d3c7c443ce43ce21a8a" gracePeriod=30
Oct 30 20:28:16.863970 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:16.863910    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6" podUID="828c3ad9-e0a7-479c-bbc6-7da1820a5ac0" containerName="main" containerID="cri-o://54b37523a5ab1e35c0010324eff11ccf17c1dea5e0347a9d92146bb36930283e" gracePeriod=45
Oct 30 20:28:16.901793 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:16.901756    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"]
Oct 30 20:28:16.911116 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:16.911078    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wrftv"]
Oct 30 20:28:17.137274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.137238    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:28:17.229353 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.229319    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-tls-certs\" (UniqueName: \"kubernetes.io/secret/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-tls-certs\") pod \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") "
Oct 30 20:28:17.229566 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.229361    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-kube-api-access-fwx28\" (UniqueName: \"kubernetes.io/projected/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-kube-api-access-fwx28\") pod \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") "
Oct 30 20:28:17.229566 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.229401    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-home\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-home\") pod \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") "
Oct 30 20:28:17.229566 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.229425    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-dshm\") pod \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") "
Oct 30 20:28:17.229566 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.229487    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-model-cache\") pod \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") "
Oct 30 20:28:17.229566 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.229526    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-tmp-dir\") pod \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\" (UID: \"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0\") "
Oct 30 20:28:17.229895 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.229793    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-home" pod "828c3ad9-e0a7-479c-bbc6-7da1820a5ac0" (UID: "828c3ad9-e0a7-479c-bbc6-7da1820a5ac0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:17.229895 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.229805    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-model-cache" pod "828c3ad9-e0a7-479c-bbc6-7da1820a5ac0" (UID: "828c3ad9-e0a7-479c-bbc6-7da1820a5ac0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:17.230022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.229918    4823 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-home\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:17.230022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.229932    4823 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-model-cache\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:17.230143 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.230067    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-tmp-dir" pod "828c3ad9-e0a7-479c-bbc6-7da1820a5ac0" (UID: "828c3ad9-e0a7-479c-bbc6-7da1820a5ac0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:17.231985 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.231957    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-kube-api-access-fwx28" pod "828c3ad9-e0a7-479c-bbc6-7da1820a5ac0" (UID: "828c3ad9-e0a7-479c-bbc6-7da1820a5ac0"). InnerVolumeSpecName "kube-api-access-fwx28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:17.232267 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.232244    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-dshm" pod "828c3ad9-e0a7-479c-bbc6-7da1820a5ac0" (UID: "828c3ad9-e0a7-479c-bbc6-7da1820a5ac0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:17.232344 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.232247    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-tls-certs" pod "828c3ad9-e0a7-479c-bbc6-7da1820a5ac0" (UID: "828c3ad9-e0a7-479c-bbc6-7da1820a5ac0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:17.330707 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.330663    4823 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-tmp-dir\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:17.330707 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.330704    4823 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-tls-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:17.330987 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.330718    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwx28\" (UniqueName: \"kubernetes.io/projected/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-kube-api-access-fwx28\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:17.330987 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.330732    4823 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0-dshm\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:17.725876 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.725830    4823 generic.go:356] "Generic (PLEG): container finished" podID="828c3ad9-e0a7-479c-bbc6-7da1820a5ac0" containerID="54b37523a5ab1e35c0010324eff11ccf17c1dea5e0347a9d92146bb36930283e" exitCode=0
Oct 30 20:28:17.726057 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.725900    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"
Oct 30 20:28:17.726057 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.725916    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6" event={"ID":"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0","Type":"ContainerDied","Data":"54b37523a5ab1e35c0010324eff11ccf17c1dea5e0347a9d92146bb36930283e"}
Oct 30 20:28:17.726057 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.725957    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6" event={"ID":"828c3ad9-e0a7-479c-bbc6-7da1820a5ac0","Type":"ContainerDied","Data":"8c2f24835f469b21ca546163c5a866d5b6c6dad6448d13c766c9b1a22815fef8"}
Oct 30 20:28:17.726057 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.725974    4823 scope.go:122] "RemoveContainer" containerID="54b37523a5ab1e35c0010324eff11ccf17c1dea5e0347a9d92146bb36930283e"
Oct 30 20:28:17.736110 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.736090    4823 scope.go:122] "RemoveContainer" containerID="54b37523a5ab1e35c0010324eff11ccf17c1dea5e0347a9d92146bb36930283e"
Oct 30 20:28:17.736416 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:28:17.736395    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b37523a5ab1e35c0010324eff11ccf17c1dea5e0347a9d92146bb36930283e\": container with ID starting with 54b37523a5ab1e35c0010324eff11ccf17c1dea5e0347a9d92146bb36930283e not found: ID does not exist" containerID="54b37523a5ab1e35c0010324eff11ccf17c1dea5e0347a9d92146bb36930283e"
Oct 30 20:28:17.736463 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.736428    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b37523a5ab1e35c0010324eff11ccf17c1dea5e0347a9d92146bb36930283e"} err="failed to get container status \"54b37523a5ab1e35c0010324eff11ccf17c1dea5e0347a9d92146bb36930283e\": rpc error: code = NotFound desc = could not find container \"54b37523a5ab1e35c0010324eff11ccf17c1dea5e0347a9d92146bb36930283e\": container with ID starting with 54b37523a5ab1e35c0010324eff11ccf17c1dea5e0347a9d92146bb36930283e not found: ID does not exist"
Oct 30 20:28:17.750460 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.750420    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"]
Oct 30 20:28:17.758382 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:17.758347    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-69ac38/simulator-single-main-69ac38-kserve-c5c6d6884-kh6g6"]
Oct 30 20:28:18.278111 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:18.278072    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="828c3ad9-e0a7-479c-bbc6-7da1820a5ac0" path="/var/lib/kubelet/pods/828c3ad9-e0a7-479c-bbc6-7da1820a5ac0/volumes"
Oct 30 20:28:18.278555 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:18.278539    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f44fa762-1df6-4c9f-bf03-9bedd1f52fc3" path="/var/lib/kubelet/pods/f44fa762-1df6-4c9f-bf03-9bedd1f52fc3/volumes"
Oct 30 20:28:20.744486 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.744441    4823 generic.go:356] "Generic (PLEG): container finished" podID="9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" containerID="325a42d960bf5bf468f229629f701d99875e6df66f587d3c7c443ce43ce21a8a" exitCode=0
Oct 30 20:28:20.744969 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.744513    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm" event={"ID":"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79","Type":"ContainerDied","Data":"325a42d960bf5bf468f229629f701d99875e6df66f587d3c7c443ce43ce21a8a"}
Oct 30 20:28:20.744969 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.744583    4823 scope.go:122] "RemoveContainer" containerID="f6538f0e26664e78d86311c63cf11ce770c309ceeb53495fc9161f0e149fdf74"
Oct 30 20:28:20.843385 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.843313    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:28:20.861750 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.861711    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-workload-socket\") pod \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") "
Oct 30 20:28:20.861995 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.861771    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-podinfo\") pod \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") "
Oct 30 20:28:20.861995 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.861828    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istiod-ca-cert\") pod \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") "
Oct 30 20:28:20.861995 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.861966    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-credential-socket\") pod \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") "
Oct 30 20:28:20.862193 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.862000    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-kube-api-access-8pz2b\" (UniqueName: \"kubernetes.io/projected/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-kube-api-access-8pz2b\") pod \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") "
Oct 30 20:28:20.862193 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.862030    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-workload-socket" pod "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" (UID: "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:20.862193 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.862046    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-envoy\") pod \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") "
Oct 30 20:28:20.862193 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.862113    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-data\") pod \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") "
Oct 30 20:28:20.862193 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.862170    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-token\" (UniqueName: \"kubernetes.io/projected/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-token\") pod \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") "
Oct 30 20:28:20.862907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.862200    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-credential-socket" pod "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" (UID: "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:20.862907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.862206    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-workload-certs\") pod \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\" (UID: \"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79\") "
Oct 30 20:28:20.862907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.862298    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istiod-ca-cert" pod "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" (UID: "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:20.862907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.862441    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-data" pod "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" (UID: "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:20.862907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.862457    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-workload-certs" pod "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" (UID: "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:20.862907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.862632    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:20.862907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.862653    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:20.862907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.862666    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:20.862907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.862681    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:20.862907 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.862694    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:20.864332 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.864288    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-kube-api-access-8pz2b" pod "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" (UID: "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79"). InnerVolumeSpecName "kube-api-access-8pz2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:20.864452 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.864432    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-token" pod "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" (UID: "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:20.864588 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.864567    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-podinfo" pod "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" (UID: "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:28:20.864648 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.864578    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-envoy" pod "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" (UID: "9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:20.963241 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.963198    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pz2b\" (UniqueName: \"kubernetes.io/projected/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-kube-api-access-8pz2b\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:20.963241 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.963235    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:20.963241 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.963245    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:20.963503 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:20.963255    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:21.752029 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:21.751998    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"
Oct 30 20:28:21.752521 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:21.752047    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm" event={"ID":"9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79","Type":"ContainerDied","Data":"3a64466f24ecbf2b526f50c9ae0cb44d9d3629ba054ff02e4745e1ee2a5d7b83"}
Oct 30 20:28:21.752521 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:21.752111    4823 scope.go:122] "RemoveContainer" containerID="325a42d960bf5bf468f229629f701d99875e6df66f587d3c7c443ce43ce21a8a"
Oct 30 20:28:21.774954 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:21.774918    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"]
Oct 30 20:28:21.777268 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:21.777234    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-69ac38-openshift-default-7fcd8d768fjfxbm"]
Oct 30 20:28:22.277597 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:22.277560    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" path="/var/lib/kubelet/pods/9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79/volumes"
Oct 30 20:28:28.499075 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:28.499035    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:28:32.019160 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:28:32.019119    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:28:32.520222 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:32.520183    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:28:48.691312 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.691273    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"]
Oct 30 20:28:48.692022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.691998    4823 cpu_manager.go:414] "containerMap: removing container" podUID="9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" containerName="istio-proxy"
Oct 30 20:28:48.692080 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.692026    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" containerName="istio-proxy"
Oct 30 20:28:48.692080 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.692042    4823 cpu_manager.go:414] "containerMap: removing container" podUID="e52412a6-bd5d-477e-af3b-12095d7f5529" containerName="revoke-keys"
Oct 30 20:28:48.692080 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.692051    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52412a6-bd5d-477e-af3b-12095d7f5529" containerName="revoke-keys"
Oct 30 20:28:48.692197 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.692086    4823 cpu_manager.go:414] "containerMap: removing container" podUID="828c3ad9-e0a7-479c-bbc6-7da1820a5ac0" containerName="main"
Oct 30 20:28:48.692197 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.692094    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828c3ad9-e0a7-479c-bbc6-7da1820a5ac0" containerName="main"
Oct 30 20:28:48.692279 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.692230    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" containerName="istio-proxy"
Oct 30 20:28:48.692279 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.692251    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52412a6-bd5d-477e-af3b-12095d7f5529" containerName="revoke-keys"
Oct 30 20:28:48.692279 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.692265    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="828c3ad9-e0a7-479c-bbc6-7da1820a5ac0" containerName="main"
Oct 30 20:28:48.695979 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.695945    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.709392 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.709355    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"]
Oct 30 20:28:48.848784 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.848738    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff945b30-c4e9-4f06-b291-821104534ee7-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.848784 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.848784    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff945b30-c4e9-4f06-b291-821104534ee7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.849071 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.848944    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff945b30-c4e9-4f06-b291-821104534ee7-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.849071 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.848975    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff945b30-c4e9-4f06-b291-821104534ee7-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.849071 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.849030    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff945b30-c4e9-4f06-b291-821104534ee7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.849071 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.849056    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff945b30-c4e9-4f06-b291-821104534ee7-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.849212 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.849131    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff945b30-c4e9-4f06-b291-821104534ee7-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.849212 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.849162    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d98pz\" (UniqueName: \"kubernetes.io/projected/ff945b30-c4e9-4f06-b291-821104534ee7-kube-api-access-d98pz\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.849281 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.849236    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff945b30-c4e9-4f06-b291-821104534ee7-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.950586 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.950495    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff945b30-c4e9-4f06-b291-821104534ee7-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.950751 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.950585    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff945b30-c4e9-4f06-b291-821104534ee7-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.950751 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.950615    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff945b30-c4e9-4f06-b291-821104534ee7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.950751 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.950669    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff945b30-c4e9-4f06-b291-821104534ee7-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.950751 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.950696    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff945b30-c4e9-4f06-b291-821104534ee7-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.950751 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.950727    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff945b30-c4e9-4f06-b291-821104534ee7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.951104 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.950760    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff945b30-c4e9-4f06-b291-821104534ee7-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.951104 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.950826    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff945b30-c4e9-4f06-b291-821104534ee7-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.951104 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.950909    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d98pz\" (UniqueName: \"kubernetes.io/projected/ff945b30-c4e9-4f06-b291-821104534ee7-kube-api-access-d98pz\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.951104 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.950992    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff945b30-c4e9-4f06-b291-821104534ee7-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.951340 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.951149    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff945b30-c4e9-4f06-b291-821104534ee7-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.951402 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.951335    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff945b30-c4e9-4f06-b291-821104534ee7-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.951470 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.951426    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff945b30-c4e9-4f06-b291-821104534ee7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.951593 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.951574    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff945b30-c4e9-4f06-b291-821104534ee7-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.953207 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.953183    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff945b30-c4e9-4f06-b291-821104534ee7-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.953318 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.953302    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff945b30-c4e9-4f06-b291-821104534ee7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.964081 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.964052    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff945b30-c4e9-4f06-b291-821104534ee7-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:48.964320 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:48.964296    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d98pz\" (UniqueName: \"kubernetes.io/projected/ff945b30-c4e9-4f06-b291-821104534ee7-kube-api-access-d98pz\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cms2d\" (UID: \"ff945b30-c4e9-4f06-b291-821104534ee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:49.013890 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:49.013841    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:49.304332 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:49.304306    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"]
Oct 30 20:28:49.308246 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:28:49.308206    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff945b30_c4e9_4f06_b291_821104534ee7.slice/crio-cced8b7f22df67c9625e88612b577e7b43aae53903bf957ad9e48d93ceaea0fa WatchSource:0}: Error finding container cced8b7f22df67c9625e88612b577e7b43aae53903bf957ad9e48d93ceaea0fa: Status 404 returned error can't find the container with id cced8b7f22df67c9625e88612b577e7b43aae53903bf957ad9e48d93ceaea0fa
Oct 30 20:28:49.312724 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:49.312689    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:28:49.312831 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:49.312773    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:28:49.312831 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:49.312804    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:28:49.929263 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:49.929218    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d" event={"ID":"ff945b30-c4e9-4f06-b291-821104534ee7","Type":"ContainerStarted","Data":"2f40a95b8d8e6b9d0483b4d8cc5d917d4afc71ecec597f35bcb60a19573d5f23"}
Oct 30 20:28:49.929690 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:49.929282    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d" event={"ID":"ff945b30-c4e9-4f06-b291-821104534ee7","Type":"ContainerStarted","Data":"cced8b7f22df67c9625e88612b577e7b43aae53903bf957ad9e48d93ceaea0fa"}
Oct 30 20:28:49.954671 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:49.954611    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d" podStartSLOduration=1.9545950250000002 podStartE2EDuration="1.954595025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:49.953143741 +0000 UTC m=+3529.845758066" watchObservedRunningTime="2026-10-30 20:28:49.954595025 +0000 UTC m=+3529.847209719"
Oct 30 20:28:50.014719 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:50.014678    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:50.020612 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:50.020581    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:50.934851 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:50.934821    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:28:50.935941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:28:50.935914    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:29:02.015881 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:29:02.015830    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:29:32.023270 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:29:32.023224    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:30:00.146203 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.146165    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9"]
Oct 30 20:30:00.146756 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.146684    4823 cpu_manager.go:414] "containerMap: removing container" podUID="9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" containerName="istio-proxy"
Oct 30 20:30:00.146756 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.146697    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" containerName="istio-proxy"
Oct 30 20:30:00.146842 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.146774    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3d8e2b-c5fb-4d10-8ff1-c69f4592df79" containerName="istio-proxy"
Oct 30 20:30:00.150286 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.150264    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9"
Oct 30 20:30:00.157982 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.157950    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:30:00.158365 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.158345    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:30:00.158556 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.158538    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:30:00.160250 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.159891    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9"]
Oct 30 20:30:00.304739 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.304691    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxngv\" (UniqueName: \"kubernetes.io/projected/5df7b00c-6302-4473-9d7b-243664652767-kube-api-access-lxngv\") pod \"maas-api-key-cleanup-29889870-tq6k9\" (UID: \"5df7b00c-6302-4473-9d7b-243664652767\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9"
Oct 30 20:30:00.304961 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.304745    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df7b00c-6302-4473-9d7b-243664652767-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-tq6k9\" (UID: \"5df7b00c-6302-4473-9d7b-243664652767\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9"
Oct 30 20:30:00.406456 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.406362    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxngv\" (UniqueName: \"kubernetes.io/projected/5df7b00c-6302-4473-9d7b-243664652767-kube-api-access-lxngv\") pod \"maas-api-key-cleanup-29889870-tq6k9\" (UID: \"5df7b00c-6302-4473-9d7b-243664652767\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9"
Oct 30 20:30:00.406456 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.406408    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df7b00c-6302-4473-9d7b-243664652767-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-tq6k9\" (UID: \"5df7b00c-6302-4473-9d7b-243664652767\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9"
Oct 30 20:30:00.407232 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.407207    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df7b00c-6302-4473-9d7b-243664652767-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-tq6k9\" (UID: \"5df7b00c-6302-4473-9d7b-243664652767\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9"
Oct 30 20:30:00.418268 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.418240    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxngv\" (UniqueName: \"kubernetes.io/projected/5df7b00c-6302-4473-9d7b-243664652767-kube-api-access-lxngv\") pod \"maas-api-key-cleanup-29889870-tq6k9\" (UID: \"5df7b00c-6302-4473-9d7b-243664652767\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9"
Oct 30 20:30:00.471223 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.471190    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:30:00.479318 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.479276    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9"
Oct 30 20:30:00.759117 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:00.759090    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9"]
Oct 30 20:30:00.762040 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:30:00.762007    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df7b00c_6302_4473_9d7b_243664652767.slice/crio-d5fc3d95315ea00c4baa4eb30e20973a20df17d336ca6406b4d3211e1dca9e2c WatchSource:0}: Error finding container d5fc3d95315ea00c4baa4eb30e20973a20df17d336ca6406b4d3211e1dca9e2c: Status 404 returned error can't find the container with id d5fc3d95315ea00c4baa4eb30e20973a20df17d336ca6406b4d3211e1dca9e2c
Oct 30 20:30:01.327020 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:01.326916    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9" event={"ID":"5df7b00c-6302-4473-9d7b-243664652767","Type":"ContainerStarted","Data":"f047860bb56abc5ee21b8e93cf3d9ddb15e03d4b332d0167eed0ba3e4c55d0b1"}
Oct 30 20:30:01.327020 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:01.326976    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9" event={"ID":"5df7b00c-6302-4473-9d7b-243664652767","Type":"ContainerStarted","Data":"d5fc3d95315ea00c4baa4eb30e20973a20df17d336ca6406b4d3211e1dca9e2c"}
Oct 30 20:30:01.345612 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:01.345558    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9" podStartSLOduration=1.345542252 podStartE2EDuration="1.345542252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:01.344003689 +0000 UTC m=+3601.236618010" watchObservedRunningTime="2026-10-30 20:30:01.345542252 +0000 UTC m=+3601.238156575"
Oct 30 20:30:01.796711 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:01.796675    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:30:01.805583 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:01.805550    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:30:02.015215 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:30:02.015157    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:30:02.333884 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:02.333816    4823 generic.go:356] "Generic (PLEG): container finished" podID="5df7b00c-6302-4473-9d7b-243664652767" containerID="f047860bb56abc5ee21b8e93cf3d9ddb15e03d4b332d0167eed0ba3e4c55d0b1" exitCode=60
Oct 30 20:30:02.334371 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:02.333901    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9" event={"ID":"5df7b00c-6302-4473-9d7b-243664652767","Type":"ContainerDied","Data":"f047860bb56abc5ee21b8e93cf3d9ddb15e03d4b332d0167eed0ba3e4c55d0b1"}
Oct 30 20:30:02.334371 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:02.334180    4823 scope.go:122] "RemoveContainer" containerID="f047860bb56abc5ee21b8e93cf3d9ddb15e03d4b332d0167eed0ba3e4c55d0b1"
Oct 30 20:30:03.341617 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:03.341583    4823 generic.go:356] "Generic (PLEG): container finished" podID="5df7b00c-6302-4473-9d7b-243664652767" containerID="e8a47a5d270797e2bf76720ce7d6fafa3e8ac9ad89015237c9b958197f903ead" exitCode=60
Oct 30 20:30:03.342067 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:03.341625    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9" event={"ID":"5df7b00c-6302-4473-9d7b-243664652767","Type":"ContainerDied","Data":"e8a47a5d270797e2bf76720ce7d6fafa3e8ac9ad89015237c9b958197f903ead"}
Oct 30 20:30:03.342067 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:03.341683    4823 scope.go:122] "RemoveContainer" containerID="f047860bb56abc5ee21b8e93cf3d9ddb15e03d4b332d0167eed0ba3e4c55d0b1"
Oct 30 20:30:03.342067 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:03.341921    4823 scope.go:122] "RemoveContainer" containerID="e8a47a5d270797e2bf76720ce7d6fafa3e8ac9ad89015237c9b958197f903ead"
Oct 30 20:30:03.342213 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:30:03.342188    4823 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-tq6k9_odh-ai-gateway-infra(5df7b00c-6302-4473-9d7b-243664652767)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9" podUID="5df7b00c-6302-4473-9d7b-243664652767"
Oct 30 20:30:03.600158 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:03.600061    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:30:04.350538 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:04.350505    4823 scope.go:122] "RemoveContainer" containerID="e8a47a5d270797e2bf76720ce7d6fafa3e8ac9ad89015237c9b958197f903ead"
Oct 30 20:30:04.350971 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:30:04.350796    4823 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-tq6k9_odh-ai-gateway-infra(5df7b00c-6302-4473-9d7b-243664652767)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9" podUID="5df7b00c-6302-4473-9d7b-243664652767"
Oct 30 20:30:14.425617 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:14.425584    4823 generic.go:356] "Generic (PLEG): container finished" podID="ff945b30-c4e9-4f06-b291-821104534ee7" containerID="2f40a95b8d8e6b9d0483b4d8cc5d917d4afc71ecec597f35bcb60a19573d5f23" exitCode=0
Oct 30 20:30:14.426078 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:14.425656    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d" event={"ID":"ff945b30-c4e9-4f06-b291-821104534ee7","Type":"ContainerDied","Data":"2f40a95b8d8e6b9d0483b4d8cc5d917d4afc71ecec597f35bcb60a19573d5f23"}
Oct 30 20:30:14.426125 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:14.426087    4823 scope.go:122] "RemoveContainer" containerID="2f40a95b8d8e6b9d0483b4d8cc5d917d4afc71ecec597f35bcb60a19573d5f23"
Oct 30 20:30:14.427301 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:14.427273    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:30:14.427369 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:14.427343    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:30:14.427409 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:14.427378    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:30:15.436108 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:15.436069    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d" event={"ID":"ff945b30-c4e9-4f06-b291-821104534ee7","Type":"ContainerStarted","Data":"fabf77633c80fcfbdd75132c6dba81083e6aaf9cced0965eee5fac71f5811f66"}
Oct 30 20:30:16.014945 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:16.014902    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:30:16.016255 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:16.016223    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cms2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 30 20:30:16.016431 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:16.016283    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d" podUID="ff945b30-c4e9-4f06-b291-821104534ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 30 20:30:17.015121 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:17.015079    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cms2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 30 20:30:17.015541 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:17.015154    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d" podUID="ff945b30-c4e9-4f06-b291-821104534ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 30 20:30:18.014538 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:18.014498    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cms2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 30 20:30:18.014744 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:18.014570    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d" podUID="ff945b30-c4e9-4f06-b291-821104534ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 30 20:30:18.278076 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:18.277980    4823 scope.go:122] "RemoveContainer" containerID="e8a47a5d270797e2bf76720ce7d6fafa3e8ac9ad89015237c9b958197f903ead"
Oct 30 20:30:18.997435 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:18.997345    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:30:19.015269 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:19.015237    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cms2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 30 20:30:19.015453 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:19.015289    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d" podUID="ff945b30-c4e9-4f06-b291-821104534ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 30 20:30:19.489097 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:19.487873    4823 generic.go:356] "Generic (PLEG): container finished" podID="5df7b00c-6302-4473-9d7b-243664652767" containerID="f6543ca0b2fffaa1eba4be899d15c2166b141620211c96d62cb32c07d1e68370" exitCode=60
Oct 30 20:30:19.489097 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:19.487950    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9" event={"ID":"5df7b00c-6302-4473-9d7b-243664652767","Type":"ContainerDied","Data":"f6543ca0b2fffaa1eba4be899d15c2166b141620211c96d62cb32c07d1e68370"}
Oct 30 20:30:19.489097 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:19.488010    4823 scope.go:122] "RemoveContainer" containerID="e8a47a5d270797e2bf76720ce7d6fafa3e8ac9ad89015237c9b958197f903ead"
Oct 30 20:30:19.489097 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:19.488520    4823 scope.go:122] "RemoveContainer" containerID="f6543ca0b2fffaa1eba4be899d15c2166b141620211c96d62cb32c07d1e68370"
Oct 30 20:30:19.489097 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:30:19.488826    4823 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-tq6k9_odh-ai-gateway-infra(5df7b00c-6302-4473-9d7b-243664652767)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9" podUID="5df7b00c-6302-4473-9d7b-243664652767"
Oct 30 20:30:20.014563 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:20.014526    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cms2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 30 20:30:20.014736 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:20.014578    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d" podUID="ff945b30-c4e9-4f06-b291-821104534ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 30 20:30:20.519941 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:20.519904    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9"]
Oct 30 20:30:20.692339 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:20.692309    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9"
Oct 30 20:30:20.784978 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:20.784851    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df7b00c-6302-4473-9d7b-243664652767-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df7b00c-6302-4473-9d7b-243664652767-maas-api-service-ca\") pod \"5df7b00c-6302-4473-9d7b-243664652767\" (UID: \"5df7b00c-6302-4473-9d7b-243664652767\") "
Oct 30 20:30:20.785176 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:20.784997    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df7b00c-6302-4473-9d7b-243664652767-kube-api-access-lxngv\" (UniqueName: \"kubernetes.io/projected/5df7b00c-6302-4473-9d7b-243664652767-kube-api-access-lxngv\") pod \"5df7b00c-6302-4473-9d7b-243664652767\" (UID: \"5df7b00c-6302-4473-9d7b-243664652767\") "
Oct 30 20:30:20.785262 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:20.785227    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df7b00c-6302-4473-9d7b-243664652767-maas-api-service-ca" pod "5df7b00c-6302-4473-9d7b-243664652767" (UID: "5df7b00c-6302-4473-9d7b-243664652767"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:20.800761 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:20.788281    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df7b00c-6302-4473-9d7b-243664652767-maas-api-service-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:30:20.800761 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:20.791191    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df7b00c-6302-4473-9d7b-243664652767-kube-api-access-lxngv" pod "5df7b00c-6302-4473-9d7b-243664652767" (UID: "5df7b00c-6302-4473-9d7b-243664652767"). InnerVolumeSpecName "kube-api-access-lxngv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:20.889570 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:20.889535    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxngv\" (UniqueName: \"kubernetes.io/projected/5df7b00c-6302-4473-9d7b-243664652767-kube-api-access-lxngv\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:30:20.935624 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:20.935580    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:30:21.015241 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:21.015200    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cms2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 30 20:30:21.015463 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:21.015257    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d" podUID="ff945b30-c4e9-4f06-b291-821104534ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 30 20:30:21.504961 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:21.504923    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9" event={"ID":"5df7b00c-6302-4473-9d7b-243664652767","Type":"ContainerDied","Data":"d5fc3d95315ea00c4baa4eb30e20973a20df17d336ca6406b4d3211e1dca9e2c"}
Oct 30 20:30:21.504961 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:21.504955    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9"
Oct 30 20:30:21.505212 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:21.504991    4823 scope.go:122] "RemoveContainer" containerID="f6543ca0b2fffaa1eba4be899d15c2166b141620211c96d62cb32c07d1e68370"
Oct 30 20:30:21.536611 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:21.536573    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9"]
Oct 30 20:30:21.542550 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:21.542520    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-tq6k9"]
Oct 30 20:30:22.014743 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:22.014694    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cms2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Oct 30 20:30:22.014975 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:22.014766    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d" podUID="ff945b30-c4e9-4f06-b291-821104534ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Oct 30 20:30:22.282199 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:22.282097    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df7b00c-6302-4473-9d7b-243664652767" path="/var/lib/kubelet/pods/5df7b00c-6302-4473-9d7b-243664652767/volumes"
Oct 30 20:30:23.402993 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:23.402959    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:30:24.019111 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:24.019063    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cms2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:30:24.019324 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:24.019136    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d" podUID="ff945b30-c4e9-4f06-b291-821104534ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:30:25.019479 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:25.019430    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cms2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:30:25.019917 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:25.019514    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d" podUID="ff945b30-c4e9-4f06-b291-821104534ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:30:26.020710 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:26.020666    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cms2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:30:26.021185 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:26.020746    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d" podUID="ff945b30-c4e9-4f06-b291-821104534ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:30:26.901679 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:26.901639    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"]
Oct 30 20:30:26.902435 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:26.902414    4823 cpu_manager.go:414] "containerMap: removing container" podUID="5df7b00c-6302-4473-9d7b-243664652767" containerName="cleanup"
Oct 30 20:30:26.902489 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:26.902439    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df7b00c-6302-4473-9d7b-243664652767" containerName="cleanup"
Oct 30 20:30:26.902489 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:26.902455    4823 cpu_manager.go:414] "containerMap: removing container" podUID="5df7b00c-6302-4473-9d7b-243664652767" containerName="cleanup"
Oct 30 20:30:26.902489 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:26.902463    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df7b00c-6302-4473-9d7b-243664652767" containerName="cleanup"
Oct 30 20:30:26.902643 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:26.902631    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df7b00c-6302-4473-9d7b-243664652767" containerName="cleanup"
Oct 30 20:30:26.902683 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:26.902655    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df7b00c-6302-4473-9d7b-243664652767" containerName="cleanup"
Oct 30 20:30:26.907782 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:26.907757    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"
Oct 30 20:30:26.911850 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:26.911824    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:30:26.912052 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:26.911902    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:30:26.913825 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:26.913792    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"]
Oct 30 20:30:26.951932 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:26.951897    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:30:26.952118 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:26.951954    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:30:27.063071 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:27.063016    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cms2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:30:27.063616 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:27.063127    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d" podUID="ff945b30-c4e9-4f06-b291-821104534ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:30:27.069401 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:27.069361    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce424581-053b-4ee8-8fa6-81af5fa86e86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-q6swp\" (UID: \"ce424581-053b-4ee8-8fa6-81af5fa86e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"
Oct 30 20:30:27.069954 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:27.069455    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce424581-053b-4ee8-8fa6-81af5fa86e86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-q6swp\" (UID: \"ce424581-053b-4ee8-8fa6-81af5fa86e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"
Oct 30 20:30:27.069954 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:27.069515    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwklw\" (UniqueName: \"kubernetes.io/projected/ce424581-053b-4ee8-8fa6-81af5fa86e86-kube-api-access-gwklw\") pod \"maas-api-revoke-sub-d2bc93bc-q6swp\" (UID: \"ce424581-053b-4ee8-8fa6-81af5fa86e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"
Oct 30 20:30:27.170533 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:27.170415    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce424581-053b-4ee8-8fa6-81af5fa86e86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-q6swp\" (UID: \"ce424581-053b-4ee8-8fa6-81af5fa86e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"
Oct 30 20:30:27.170533 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:27.170498    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce424581-053b-4ee8-8fa6-81af5fa86e86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-q6swp\" (UID: \"ce424581-053b-4ee8-8fa6-81af5fa86e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"
Oct 30 20:30:27.170803 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:27.170569    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwklw\" (UniqueName: \"kubernetes.io/projected/ce424581-053b-4ee8-8fa6-81af5fa86e86-kube-api-access-gwklw\") pod \"maas-api-revoke-sub-d2bc93bc-q6swp\" (UID: \"ce424581-053b-4ee8-8fa6-81af5fa86e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"
Oct 30 20:30:27.171440 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:27.171406    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce424581-053b-4ee8-8fa6-81af5fa86e86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-q6swp\" (UID: \"ce424581-053b-4ee8-8fa6-81af5fa86e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"
Oct 30 20:30:27.173203 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:27.173174    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce424581-053b-4ee8-8fa6-81af5fa86e86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-q6swp\" (UID: \"ce424581-053b-4ee8-8fa6-81af5fa86e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"
Oct 30 20:30:27.183120 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:27.183084    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwklw\" (UniqueName: \"kubernetes.io/projected/ce424581-053b-4ee8-8fa6-81af5fa86e86-kube-api-access-gwklw\") pod \"maas-api-revoke-sub-d2bc93bc-q6swp\" (UID: \"ce424581-053b-4ee8-8fa6-81af5fa86e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"
Oct 30 20:30:27.226949 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:27.226896    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"
Oct 30 20:30:27.527749 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:27.527719    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"]
Oct 30 20:30:27.530415 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:30:27.530373    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce424581_053b_4ee8_8fa6_81af5fa86e86.slice/crio-ef3e1a00247606a6bf179dcee656d93aa10106389b53a4e68b8d533e171591dc WatchSource:0}: Error finding container ef3e1a00247606a6bf179dcee656d93aa10106389b53a4e68b8d533e171591dc: Status 404 returned error can't find the container with id ef3e1a00247606a6bf179dcee656d93aa10106389b53a4e68b8d533e171591dc
Oct 30 20:30:27.530936 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:27.530909    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:30:27.532162 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:27.532140    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cms2d"
Oct 30 20:30:27.551518 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:27.551486    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp" event={"ID":"ce424581-053b-4ee8-8fa6-81af5fa86e86","Type":"ContainerStarted","Data":"ef3e1a00247606a6bf179dcee656d93aa10106389b53a4e68b8d533e171591dc"}
Oct 30 20:30:28.558998 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:28.558956    4823 generic.go:356] "Generic (PLEG): container finished" podID="ce424581-053b-4ee8-8fa6-81af5fa86e86" containerID="8027717e5a3b749e5892f81d410d3a4a2035a04bff209e13fc16db576f6b678a" exitCode=0
Oct 30 20:30:28.559382 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:28.559049    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp" event={"ID":"ce424581-053b-4ee8-8fa6-81af5fa86e86","Type":"ContainerDied","Data":"8027717e5a3b749e5892f81d410d3a4a2035a04bff209e13fc16db576f6b678a"}
Oct 30 20:30:29.740598 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:29.740566    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"
Oct 30 20:30:29.899832 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:29.899799    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce424581-053b-4ee8-8fa6-81af5fa86e86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce424581-053b-4ee8-8fa6-81af5fa86e86-maas-api-xks-ca\") pod \"ce424581-053b-4ee8-8fa6-81af5fa86e86\" (UID: \"ce424581-053b-4ee8-8fa6-81af5fa86e86\") "
Oct 30 20:30:29.899832 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:29.899847    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce424581-053b-4ee8-8fa6-81af5fa86e86-kube-api-access-gwklw\" (UniqueName: \"kubernetes.io/projected/ce424581-053b-4ee8-8fa6-81af5fa86e86-kube-api-access-gwklw\") pod \"ce424581-053b-4ee8-8fa6-81af5fa86e86\" (UID: \"ce424581-053b-4ee8-8fa6-81af5fa86e86\") "
Oct 30 20:30:29.900138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:29.899910    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce424581-053b-4ee8-8fa6-81af5fa86e86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce424581-053b-4ee8-8fa6-81af5fa86e86-maas-api-openshift-ca\") pod \"ce424581-053b-4ee8-8fa6-81af5fa86e86\" (UID: \"ce424581-053b-4ee8-8fa6-81af5fa86e86\") "
Oct 30 20:30:29.900344 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:29.900316    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce424581-053b-4ee8-8fa6-81af5fa86e86-maas-api-openshift-ca" pod "ce424581-053b-4ee8-8fa6-81af5fa86e86" (UID: "ce424581-053b-4ee8-8fa6-81af5fa86e86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:29.902107 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:29.902077    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce424581-053b-4ee8-8fa6-81af5fa86e86-kube-api-access-gwklw" pod "ce424581-053b-4ee8-8fa6-81af5fa86e86" (UID: "ce424581-053b-4ee8-8fa6-81af5fa86e86"). InnerVolumeSpecName "kube-api-access-gwklw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:29.902241 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:29.902146    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce424581-053b-4ee8-8fa6-81af5fa86e86-maas-api-xks-ca" pod "ce424581-053b-4ee8-8fa6-81af5fa86e86" (UID: "ce424581-053b-4ee8-8fa6-81af5fa86e86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:30.001465 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:30.001427    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce424581-053b-4ee8-8fa6-81af5fa86e86-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:30:30.001465 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:30.001462    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwklw\" (UniqueName: \"kubernetes.io/projected/ce424581-053b-4ee8-8fa6-81af5fa86e86-kube-api-access-gwklw\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:30:30.001465 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:30.001473    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce424581-053b-4ee8-8fa6-81af5fa86e86-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:30:30.572014 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:30.571925    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"
Oct 30 20:30:30.572014 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:30.571960    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp" event={"ID":"ce424581-053b-4ee8-8fa6-81af5fa86e86","Type":"ContainerDied","Data":"ef3e1a00247606a6bf179dcee656d93aa10106389b53a4e68b8d533e171591dc"}
Oct 30 20:30:30.572014 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:30.572007    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef3e1a00247606a6bf179dcee656d93aa10106389b53a4e68b8d533e171591dc"
Oct 30 20:30:32.015521 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:30:32.015475    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:30:37.812845 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:37.812803    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:30:38.969014 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:38.968970    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"]
Oct 30 20:30:38.969787 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:38.969761    4823 cpu_manager.go:414] "containerMap: removing container" podUID="ce424581-053b-4ee8-8fa6-81af5fa86e86" containerName="revoke-keys"
Oct 30 20:30:38.969992 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:38.969808    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce424581-053b-4ee8-8fa6-81af5fa86e86" containerName="revoke-keys"
Oct 30 20:30:38.969992 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:38.969825    4823 cpu_manager.go:414] "containerMap: removing container" podUID="5df7b00c-6302-4473-9d7b-243664652767" containerName="cleanup"
Oct 30 20:30:38.969992 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:38.969834    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df7b00c-6302-4473-9d7b-243664652767" containerName="cleanup"
Oct 30 20:30:38.970127 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:38.970008    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce424581-053b-4ee8-8fa6-81af5fa86e86" containerName="revoke-keys"
Oct 30 20:30:38.970127 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:38.970029    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df7b00c-6302-4473-9d7b-243664652767" containerName="cleanup"
Oct 30 20:30:38.972836 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:38.972816    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:38.977311 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:38.977284    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-de61d2-openshift-default-dockercfg-fk9qc\""
Oct 30 20:30:38.993516 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:38.993461    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"]
Oct 30 20:30:39.093489 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.093439    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-credential-socket\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.093693 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.093496    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78af1771-9443-4d67-a451-e58862d2ef22-istio-token\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.093693 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.093674    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-workload-socket\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.093815 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.093714    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-istio-data\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.093906 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.093810    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-istio-envoy\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.093906 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.093851    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78af1771-9443-4d67-a451-e58862d2ef22-istiod-ca-cert\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.094033 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.093969    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qndbh\" (UniqueName: \"kubernetes.io/projected/78af1771-9443-4d67-a451-e58862d2ef22-kube-api-access-qndbh\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.094103 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.094080    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78af1771-9443-4d67-a451-e58862d2ef22-istio-podinfo\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.094164 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.094142    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-workload-certs\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.194925 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.194846    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78af1771-9443-4d67-a451-e58862d2ef22-istio-podinfo\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.195138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.194937    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-workload-certs\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.195138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.194984    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-credential-socket\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.195138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.195001    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78af1771-9443-4d67-a451-e58862d2ef22-istio-token\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.195138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.195028    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-workload-socket\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.195138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.195049    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-istio-data\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.195138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.195083    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-istio-envoy\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.195138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.195108    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78af1771-9443-4d67-a451-e58862d2ef22-istiod-ca-cert\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.195542 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.195159    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndbh\" (UniqueName: \"kubernetes.io/projected/78af1771-9443-4d67-a451-e58862d2ef22-kube-api-access-qndbh\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.195542 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.195489    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-istio-data\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.195542 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.195514    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-workload-certs\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.195718 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.195583    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-workload-socket\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.195830 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.195811    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-credential-socket\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.196004 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.195981    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78af1771-9443-4d67-a451-e58862d2ef22-istiod-ca-cert\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.197508 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.197484    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-istio-envoy\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.197732 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.197710    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78af1771-9443-4d67-a451-e58862d2ef22-istio-podinfo\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.207749 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.207721    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78af1771-9443-4d67-a451-e58862d2ef22-istio-token\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.209043 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.209018    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qndbh\" (UniqueName: \"kubernetes.io/projected/78af1771-9443-4d67-a451-e58862d2ef22-kube-api-access-qndbh\") pod \"e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") " pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.286226 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.286140    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:39.577105 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.577078    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"]
Oct 30 20:30:39.579042 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:30:39.579007    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78af1771_9443_4d67_a451_e58862d2ef22.slice/crio-457eccfdf2fe3a524054a3c1ec3181ee044845c43d68f7428128f66fb2d33b79 WatchSource:0}: Error finding container 457eccfdf2fe3a524054a3c1ec3181ee044845c43d68f7428128f66fb2d33b79: Status 404 returned error can't find the container with id 457eccfdf2fe3a524054a3c1ec3181ee044845c43d68f7428128f66fb2d33b79
Oct 30 20:30:39.581208 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.581169    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:30:39.581316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.581243    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:30:39.581316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.581276    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:30:39.624094 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:39.624055    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2" event={"ID":"78af1771-9443-4d67-a451-e58862d2ef22","Type":"ContainerStarted","Data":"457eccfdf2fe3a524054a3c1ec3181ee044845c43d68f7428128f66fb2d33b79"}
Oct 30 20:30:40.631598 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:40.631509    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2" event={"ID":"78af1771-9443-4d67-a451-e58862d2ef22","Type":"ContainerStarted","Data":"8a2d7e137c6e0d6c5e1268df9ef742a985585d6aaabb6fb39c994fd81003271c"}
Oct 30 20:30:40.660247 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:40.660179    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2" podStartSLOduration=2.6601607229999997 podStartE2EDuration="2.660160723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:40.656041909 +0000 UTC m=+3640.548656234" watchObservedRunningTime="2026-10-30 20:30:40.660160723 +0000 UTC m=+3640.552775046"
Oct 30 20:30:41.286602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:41.286568    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:41.292031 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:41.291997    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:41.638588 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:41.638557    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:41.639604 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:41.639582    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:44.106767 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.106725    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"]
Oct 30 20:30:44.110527 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.110498    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:30:44.120364 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.120335    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-de61d2-serving-cert\""
Oct 30 20:30:44.122758 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.122686    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47lpr\""
Oct 30 20:30:44.122758 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.122750    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:30:44.123015 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.122750    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:30:44.123015 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.122843    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 20:30:44.127004 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.126980    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"]
Oct 30 20:30:44.140084 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.140034    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9946f082-1e5d-4b13-adfa-5cb998638e4d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq\" (UID: \"9946f082-1e5d-4b13-adfa-5cb998638e4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:30:44.140445 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.140410    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9946f082-1e5d-4b13-adfa-5cb998638e4d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq\" (UID: \"9946f082-1e5d-4b13-adfa-5cb998638e4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:30:44.140568 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.140466    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsnr6\" (UniqueName: \"kubernetes.io/projected/9946f082-1e5d-4b13-adfa-5cb998638e4d-kube-api-access-nsnr6\") pod \"maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq\" (UID: \"9946f082-1e5d-4b13-adfa-5cb998638e4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:30:44.140568 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.140547    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9946f082-1e5d-4b13-adfa-5cb998638e4d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq\" (UID: \"9946f082-1e5d-4b13-adfa-5cb998638e4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:30:44.241401 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.241367    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9946f082-1e5d-4b13-adfa-5cb998638e4d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq\" (UID: \"9946f082-1e5d-4b13-adfa-5cb998638e4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:30:44.241619 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.241467    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9946f082-1e5d-4b13-adfa-5cb998638e4d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq\" (UID: \"9946f082-1e5d-4b13-adfa-5cb998638e4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:30:44.241619 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.241499    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsnr6\" (UniqueName: \"kubernetes.io/projected/9946f082-1e5d-4b13-adfa-5cb998638e4d-kube-api-access-nsnr6\") pod \"maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq\" (UID: \"9946f082-1e5d-4b13-adfa-5cb998638e4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:30:44.241619 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.241560    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9946f082-1e5d-4b13-adfa-5cb998638e4d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq\" (UID: \"9946f082-1e5d-4b13-adfa-5cb998638e4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:30:44.242466 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.242431    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9946f082-1e5d-4b13-adfa-5cb998638e4d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq\" (UID: \"9946f082-1e5d-4b13-adfa-5cb998638e4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:30:44.244113 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.244085    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9946f082-1e5d-4b13-adfa-5cb998638e4d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq\" (UID: \"9946f082-1e5d-4b13-adfa-5cb998638e4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:30:44.244113 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.244106    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9946f082-1e5d-4b13-adfa-5cb998638e4d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq\" (UID: \"9946f082-1e5d-4b13-adfa-5cb998638e4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:30:44.257883 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.257826    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsnr6\" (UniqueName: \"kubernetes.io/projected/9946f082-1e5d-4b13-adfa-5cb998638e4d-kube-api-access-nsnr6\") pod \"maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq\" (UID: \"9946f082-1e5d-4b13-adfa-5cb998638e4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:30:44.434383 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.434275    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:30:44.720624 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.720592    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"]
Oct 30 20:30:44.724506 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:30:44.724471    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9946f082_1e5d_4b13_adfa_5cb998638e4d.slice/crio-3b7a40e1d01a3bd21c284f7392b579569731e3ca7e6075c6af14692478ddd9ad WatchSource:0}: Error finding container 3b7a40e1d01a3bd21c284f7392b579569731e3ca7e6075c6af14692478ddd9ad: Status 404 returned error can't find the container with id 3b7a40e1d01a3bd21c284f7392b579569731e3ca7e6075c6af14692478ddd9ad
Oct 30 20:30:44.726435 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:44.726411    4823 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:30:45.673370 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:45.673327    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq" event={"ID":"9946f082-1e5d-4b13-adfa-5cb998638e4d","Type":"ContainerStarted","Data":"f568cd048cf07927b5e640a777c3e23d8212a3ff2b45f46de5a176856f73d9b0"}
Oct 30 20:30:45.673763 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:45.673416    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq" event={"ID":"9946f082-1e5d-4b13-adfa-5cb998638e4d","Type":"ContainerStarted","Data":"3b7a40e1d01a3bd21c284f7392b579569731e3ca7e6075c6af14692478ddd9ad"}
Oct 30 20:30:45.673763 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:45.673521    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:30:45.694136 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:45.694085    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq" podStartSLOduration=1.114569216 podStartE2EDuration="1.694048169s" totalImagesPullingTime="579.478953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:45.691760639 +0000 UTC m=+3645.584374963" watchObservedRunningTime="2026-10-30 20:30:45.694048169 +0000 UTC m=+3645.586662510"
Oct 30 20:30:46.285496 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:46.285459    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:30:48.829055 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.829018    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"]
Oct 30 20:30:48.833221 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.833198    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.844321 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.844295    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"]
Oct 30 20:30:48.895022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.894984    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.895022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.895028    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.895309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.895052    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.895309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.895123    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdnqp\" (UniqueName: \"kubernetes.io/projected/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-kube-api-access-sdnqp\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.895309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.895219    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.895309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.895276    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.895545 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.895316    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.895545 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.895406    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.895545 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.895453    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.996807 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.996771    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.997009 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.996825    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.997009 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.996943    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.997009 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.996992    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.997184 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.997021    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.997184 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.997046    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.997184 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.997063    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdnqp\" (UniqueName: \"kubernetes.io/projected/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-kube-api-access-sdnqp\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.997184 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.997106    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.997184 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.997161    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.997470 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.997381    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.997470 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.997436    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.997556 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.997506    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.997750 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.997728    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.997794 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.997783    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.999494 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.999468    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:48.999631 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:48.999477    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:49.009033 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:49.009004    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdnqp\" (UniqueName: \"kubernetes.io/projected/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-kube-api-access-sdnqp\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:49.009783 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:49.009762    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e154c2-3f3b-49fb-ba0f-4c2be5d2388f-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-6kxj5\" (UID: \"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:49.148076 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:49.148041    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:49.480900 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:49.480553    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"]
Oct 30 20:30:49.481087 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:30:49.481055    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e154c2_3f3b_49fb_ba0f_4c2be5d2388f.slice/crio-070077d3145fac444d6d40470e166035e858bb3b5aa54555b3ab7c2a5a3070eb WatchSource:0}: Error finding container 070077d3145fac444d6d40470e166035e858bb3b5aa54555b3ab7c2a5a3070eb: Status 404 returned error can't find the container with id 070077d3145fac444d6d40470e166035e858bb3b5aa54555b3ab7c2a5a3070eb
Oct 30 20:30:49.484542 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:49.484505    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:30:49.484654 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:49.484592    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:30:49.484654 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:49.484636    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:30:49.701453 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:49.701417    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5" event={"ID":"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f","Type":"ContainerStarted","Data":"55a7999ba6281930888e874c0e618304f7513ab52126719169da53cb25f9bacc"}
Oct 30 20:30:49.701628 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:49.701484    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5" event={"ID":"91e154c2-3f3b-49fb-ba0f-4c2be5d2388f","Type":"ContainerStarted","Data":"070077d3145fac444d6d40470e166035e858bb3b5aa54555b3ab7c2a5a3070eb"}
Oct 30 20:30:49.728059 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:49.727965    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5" podStartSLOduration=1.727950129 podStartE2EDuration="1.727950129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:49.726084282 +0000 UTC m=+3649.618698609" watchObservedRunningTime="2026-10-30 20:30:49.727950129 +0000 UTC m=+3649.620564529"
Oct 30 20:30:50.149341 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:50.149303    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:50.150556 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:50.150530    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-6kxj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 20:30:50.150632 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:50.150583    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5" podUID="91e154c2-3f3b-49fb-ba0f-4c2be5d2388f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 20:30:51.148408 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:51.148373    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-6kxj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 20:30:51.148594 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:51.148427    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5" podUID="91e154c2-3f3b-49fb-ba0f-4c2be5d2388f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 20:30:51.702064 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:51.702023    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:30:52.149076 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:52.149040    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-6kxj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 20:30:52.149243 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:52.149094    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5" podUID="91e154c2-3f3b-49fb-ba0f-4c2be5d2388f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 20:30:53.148689 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:53.148653    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-6kxj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 20:30:53.149152 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:53.148720    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5" podUID="91e154c2-3f3b-49fb-ba0f-4c2be5d2388f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 20:30:54.149106 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:54.149064    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-6kxj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 20:30:54.149601 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:54.149117    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5" podUID="91e154c2-3f3b-49fb-ba0f-4c2be5d2388f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 20:30:55.149421 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:55.149376    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-6kxj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 20:30:55.150021 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:55.149435    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5" podUID="91e154c2-3f3b-49fb-ba0f-4c2be5d2388f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 20:30:56.149335 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:56.149290    4823 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-6kxj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 20:30:56.149522 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:56.149382    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5" podUID="91e154c2-3f3b-49fb-ba0f-4c2be5d2388f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 20:30:57.013165 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:57.013124    4823 patch_prober.go:29] interesting pod/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": read tcp 10.132.0.2:52704->10.132.0.245:15021: read: connection reset by peer" start-of-body=
Oct 30 20:30:57.013363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:57.013198    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2" podUID="78af1771-9443-4d67-a451-e58862d2ef22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": read tcp 10.132.0.2:52704->10.132.0.245:15021: read: connection reset by peer"
Oct 30 20:30:57.758850 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:57.758817    4823 generic.go:356] "Generic (PLEG): container finished" podID="78af1771-9443-4d67-a451-e58862d2ef22" containerID="8a2d7e137c6e0d6c5e1268df9ef742a985585d6aaabb6fb39c994fd81003271c" exitCode=0
Oct 30 20:30:57.759339 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:57.758903    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2" event={"ID":"78af1771-9443-4d67-a451-e58862d2ef22","Type":"ContainerDied","Data":"8a2d7e137c6e0d6c5e1268df9ef742a985585d6aaabb6fb39c994fd81003271c"}
Oct 30 20:30:57.759459 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:57.759439    4823 scope.go:122] "RemoveContainer" containerID="8a2d7e137c6e0d6c5e1268df9ef742a985585d6aaabb6fb39c994fd81003271c"
Oct 30 20:30:57.760785 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:57.760756    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:30:57.760921 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:57.760836    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:30:57.760921 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:57.760907    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:30:57.993306 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:57.993276    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:57.993595 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:57.993577    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:57.995138 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:57.995084    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-6kxj5"
Oct 30 20:30:58.783176 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:58.783132    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2" event={"ID":"78af1771-9443-4d67-a451-e58862d2ef22","Type":"ContainerStarted","Data":"7495441fb837316f1975bca972a32cb4e933d17499cc31fdae82543299e5acca"}
Oct 30 20:30:59.287215 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:59.287173    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:30:59.287427 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:59.287403    4823 patch_prober.go:29] interesting pod/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Oct 30 20:30:59.287481 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:30:59.287451    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2" podUID="78af1771-9443-4d67-a451-e58862d2ef22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Oct 30 20:31:00.289023 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:00.286698    4823 patch_prober.go:29] interesting pod/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Oct 30 20:31:00.289023 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:00.286776    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2" podUID="78af1771-9443-4d67-a451-e58862d2ef22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Oct 30 20:31:01.287387 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:01.287344    4823 patch_prober.go:29] interesting pod/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Oct 30 20:31:01.287576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:01.287405    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2" podUID="78af1771-9443-4d67-a451-e58862d2ef22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Oct 30 20:31:01.941130 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:01.941091    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:31:02.015101 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:31:02.015063    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:31:02.286839 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:02.286741    4823 patch_prober.go:29] interesting pod/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Oct 30 20:31:02.286839 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:02.286802    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2" podUID="78af1771-9443-4d67-a451-e58862d2ef22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Oct 30 20:31:03.287537 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:03.287402    4823 patch_prober.go:29] interesting pod/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body=
Oct 30 20:31:03.287537 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:03.287457    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2" podUID="78af1771-9443-4d67-a451-e58862d2ef22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused"
Oct 30 20:31:04.378923 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:04.378857    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:31:04.379469 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:04.379395    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:31:04.380166 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:04.380147    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:31:18.982381 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:18.982346    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:31:32.014007 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:31:32.013962    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:31:41.622739 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:41.622704    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:31:51.806295 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:51.806261    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"]
Oct 30 20:31:51.806774 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:51.806497    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq" podUID="9946f082-1e5d-4b13-adfa-5cb998638e4d" containerName="maas-api" containerID="cri-o://f568cd048cf07927b5e640a777c3e23d8212a3ff2b45f46de5a176856f73d9b0" gracePeriod=30
Oct 30 20:31:52.097734 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.097705    4823 generic.go:356] "Generic (PLEG): container finished" podID="9946f082-1e5d-4b13-adfa-5cb998638e4d" containerID="f568cd048cf07927b5e640a777c3e23d8212a3ff2b45f46de5a176856f73d9b0" exitCode=0
Oct 30 20:31:52.097911 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.097785    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq" event={"ID":"9946f082-1e5d-4b13-adfa-5cb998638e4d","Type":"ContainerDied","Data":"f568cd048cf07927b5e640a777c3e23d8212a3ff2b45f46de5a176856f73d9b0"}
Oct 30 20:31:52.097911 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.097839    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq" event={"ID":"9946f082-1e5d-4b13-adfa-5cb998638e4d","Type":"ContainerDied","Data":"3b7a40e1d01a3bd21c284f7392b579569731e3ca7e6075c6af14692478ddd9ad"}
Oct 30 20:31:52.097911 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.097902    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b7a40e1d01a3bd21c284f7392b579569731e3ca7e6075c6af14692478ddd9ad"
Oct 30 20:31:52.100255 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.100235    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:31:52.138208 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.138157    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9946f082-1e5d-4b13-adfa-5cb998638e4d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9946f082-1e5d-4b13-adfa-5cb998638e4d-maas-api-tls\") pod \"9946f082-1e5d-4b13-adfa-5cb998638e4d\" (UID: \"9946f082-1e5d-4b13-adfa-5cb998638e4d\") "
Oct 30 20:31:52.138412 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.138242    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9946f082-1e5d-4b13-adfa-5cb998638e4d-kube-api-access-nsnr6\" (UniqueName: \"kubernetes.io/projected/9946f082-1e5d-4b13-adfa-5cb998638e4d-kube-api-access-nsnr6\") pod \"9946f082-1e5d-4b13-adfa-5cb998638e4d\" (UID: \"9946f082-1e5d-4b13-adfa-5cb998638e4d\") "
Oct 30 20:31:52.138412 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.138332    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9946f082-1e5d-4b13-adfa-5cb998638e4d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9946f082-1e5d-4b13-adfa-5cb998638e4d-openshift-service-ca\") pod \"9946f082-1e5d-4b13-adfa-5cb998638e4d\" (UID: \"9946f082-1e5d-4b13-adfa-5cb998638e4d\") "
Oct 30 20:31:52.138412 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.138379    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9946f082-1e5d-4b13-adfa-5cb998638e4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9946f082-1e5d-4b13-adfa-5cb998638e4d-metrics-certs\") pod \"9946f082-1e5d-4b13-adfa-5cb998638e4d\" (UID: \"9946f082-1e5d-4b13-adfa-5cb998638e4d\") "
Oct 30 20:31:52.139397 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.139361    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9946f082-1e5d-4b13-adfa-5cb998638e4d-openshift-service-ca" pod "9946f082-1e5d-4b13-adfa-5cb998638e4d" (UID: "9946f082-1e5d-4b13-adfa-5cb998638e4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:52.141056 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.141028    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9946f082-1e5d-4b13-adfa-5cb998638e4d-maas-api-tls" pod "9946f082-1e5d-4b13-adfa-5cb998638e4d" (UID: "9946f082-1e5d-4b13-adfa-5cb998638e4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:52.141056 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.141044    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9946f082-1e5d-4b13-adfa-5cb998638e4d-kube-api-access-nsnr6" pod "9946f082-1e5d-4b13-adfa-5cb998638e4d" (UID: "9946f082-1e5d-4b13-adfa-5cb998638e4d"). InnerVolumeSpecName "kube-api-access-nsnr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:52.141256 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.141181    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9946f082-1e5d-4b13-adfa-5cb998638e4d-metrics-certs" pod "9946f082-1e5d-4b13-adfa-5cb998638e4d" (UID: "9946f082-1e5d-4b13-adfa-5cb998638e4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:52.240516 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.240476    4823 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9946f082-1e5d-4b13-adfa-5cb998638e4d-openshift-service-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:31:52.240516 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.240512    4823 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9946f082-1e5d-4b13-adfa-5cb998638e4d-metrics-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:31:52.240516 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.240521    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9946f082-1e5d-4b13-adfa-5cb998638e4d-maas-api-tls\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:31:52.240782 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:52.240529    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsnr6\" (UniqueName: \"kubernetes.io/projected/9946f082-1e5d-4b13-adfa-5cb998638e4d-kube-api-access-nsnr6\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:31:53.103074 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:53.103041    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"
Oct 30 20:31:53.121995 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:53.121965    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"]
Oct 30 20:31:53.132806 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:53.132773    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-de61d2-74496fcdfd-fl2nq"]
Oct 30 20:31:54.278304 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:54.278267    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9946f082-1e5d-4b13-adfa-5cb998638e4d" path="/var/lib/kubelet/pods/9946f082-1e5d-4b13-adfa-5cb998638e4d/volumes"
Oct 30 20:31:54.334935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:31:54.334898    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:32:00.008989 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:00.008954    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"]
Oct 30 20:32:00.009508 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:00.009227    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2" podUID="78af1771-9443-4d67-a451-e58862d2ef22" containerName="istio-proxy" containerID="cri-o://7495441fb837316f1975bca972a32cb4e933d17499cc31fdae82543299e5acca" gracePeriod=30
Oct 30 20:32:02.018963 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:32:02.018925    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:32:04.381396 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:04.381354    4823 patch_prober.go:29] interesting pod/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:32:04.381917 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:04.381413    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2" podUID="78af1771-9443-4d67-a451-e58862d2ef22" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:32:04.955260 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:04.955221    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"]
Oct 30 20:32:04.955821 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:04.955790    4823 cpu_manager.go:414] "containerMap: removing container" podUID="9946f082-1e5d-4b13-adfa-5cb998638e4d" containerName="maas-api"
Oct 30 20:32:04.955821 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:04.955817    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9946f082-1e5d-4b13-adfa-5cb998638e4d" containerName="maas-api"
Oct 30 20:32:04.956085 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:04.956012    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="9946f082-1e5d-4b13-adfa-5cb998638e4d" containerName="maas-api"
Oct 30 20:32:04.959365 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:04.959347    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:04.967391 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:04.966978    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-381a9150-kserve-self-signed-certs\""
Oct 30 20:32:04.971133 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:04.971112    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"]
Oct 30 20:32:05.074398 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.074368    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-model-cache\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.074532 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.074408    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-home\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.074585 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.074534    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-dshm\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.074648 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.074581    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-tmp-dir\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.074726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.074699    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbrfm\" (UniqueName: \"kubernetes.io/projected/5b56c575-497e-4524-98cc-9a0769ebc285-kube-api-access-fbrfm\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.074788 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.074764    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b56c575-497e-4524-98cc-9a0769ebc285-tls-certs\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.175563 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.175523    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbrfm\" (UniqueName: \"kubernetes.io/projected/5b56c575-497e-4524-98cc-9a0769ebc285-kube-api-access-fbrfm\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.175753 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.175603    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b56c575-497e-4524-98cc-9a0769ebc285-tls-certs\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.175753 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.175683    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-model-cache\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.175753 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.175730    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-home\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.175945 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.175826    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-dshm\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.175945 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.175918    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-tmp-dir\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.176144 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.176114    4823 generic.go:356] "Generic (PLEG): container finished" podID="78af1771-9443-4d67-a451-e58862d2ef22" containerID="7495441fb837316f1975bca972a32cb4e933d17499cc31fdae82543299e5acca" exitCode=0
Oct 30 20:32:05.176226 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.176167    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-model-cache\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.176226 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.176199    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-home\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.176343 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.176244    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2" event={"ID":"78af1771-9443-4d67-a451-e58862d2ef22","Type":"ContainerDied","Data":"7495441fb837316f1975bca972a32cb4e933d17499cc31fdae82543299e5acca"}
Oct 30 20:32:05.176343 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.176283    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-tmp-dir\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.176343 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.176295    4823 scope.go:122] "RemoveContainer" containerID="8a2d7e137c6e0d6c5e1268df9ef742a985585d6aaabb6fb39c994fd81003271c"
Oct 30 20:32:05.178637 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.178517    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-dshm\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.178757 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.178735    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b56c575-497e-4524-98cc-9a0769ebc285-tls-certs\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.188373 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.188345    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbrfm\" (UniqueName: \"kubernetes.io/projected/5b56c575-497e-4524-98cc-9a0769ebc285-kube-api-access-fbrfm\") pod \"e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") " pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.272159 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.272134    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:05.284280 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.284261    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:32:05.377926 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.377894    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78af1771-9443-4d67-a451-e58862d2ef22-kube-api-access-qndbh\" (UniqueName: \"kubernetes.io/projected/78af1771-9443-4d67-a451-e58862d2ef22-kube-api-access-qndbh\") pod \"78af1771-9443-4d67-a451-e58862d2ef22\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") "
Oct 30 20:32:05.378111 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.377973    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-istio-data\") pod \"78af1771-9443-4d67-a451-e58862d2ef22\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") "
Oct 30 20:32:05.378111 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.378020    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78af1771-9443-4d67-a451-e58862d2ef22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78af1771-9443-4d67-a451-e58862d2ef22-istiod-ca-cert\") pod \"78af1771-9443-4d67-a451-e58862d2ef22\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") "
Oct 30 20:32:05.378111 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.378056    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-workload-socket\") pod \"78af1771-9443-4d67-a451-e58862d2ef22\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") "
Oct 30 20:32:05.378111 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.378102    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-istio-envoy\") pod \"78af1771-9443-4d67-a451-e58862d2ef22\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") "
Oct 30 20:32:05.378347 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.378129    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-workload-certs\") pod \"78af1771-9443-4d67-a451-e58862d2ef22\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") "
Oct 30 20:32:05.378347 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.378199    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78af1771-9443-4d67-a451-e58862d2ef22-istio-token\" (UniqueName: \"kubernetes.io/projected/78af1771-9443-4d67-a451-e58862d2ef22-istio-token\") pod \"78af1771-9443-4d67-a451-e58862d2ef22\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") "
Oct 30 20:32:05.378347 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.378257    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-credential-socket\") pod \"78af1771-9443-4d67-a451-e58862d2ef22\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") "
Oct 30 20:32:05.378347 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.378335    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78af1771-9443-4d67-a451-e58862d2ef22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78af1771-9443-4d67-a451-e58862d2ef22-istio-podinfo\") pod \"78af1771-9443-4d67-a451-e58862d2ef22\" (UID: \"78af1771-9443-4d67-a451-e58862d2ef22\") "
Oct 30 20:32:05.378347 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.378331    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-istio-data" pod "78af1771-9443-4d67-a451-e58862d2ef22" (UID: "78af1771-9443-4d67-a451-e58862d2ef22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:32:05.378614 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.378552    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78af1771-9443-4d67-a451-e58862d2ef22-istiod-ca-cert" pod "78af1771-9443-4d67-a451-e58862d2ef22" (UID: "78af1771-9443-4d67-a451-e58862d2ef22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:32:05.378614 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.378560    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-workload-certs" pod "78af1771-9443-4d67-a451-e58862d2ef22" (UID: "78af1771-9443-4d67-a451-e58862d2ef22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:32:05.378734 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.378682    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-credential-socket" pod "78af1771-9443-4d67-a451-e58862d2ef22" (UID: "78af1771-9443-4d67-a451-e58862d2ef22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:32:05.378734 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.378702    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-workload-socket" pod "78af1771-9443-4d67-a451-e58862d2ef22" (UID: "78af1771-9443-4d67-a451-e58862d2ef22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:32:05.379100 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.379076    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:05.379209 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.379105    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:05.379209 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.379122    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78af1771-9443-4d67-a451-e58862d2ef22-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:05.379209 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.379135    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:05.379209 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.379151    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:05.380671 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.380640    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-istio-envoy" pod "78af1771-9443-4d67-a451-e58862d2ef22" (UID: "78af1771-9443-4d67-a451-e58862d2ef22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:32:05.381033 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.380999    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78af1771-9443-4d67-a451-e58862d2ef22-istio-token" pod "78af1771-9443-4d67-a451-e58862d2ef22" (UID: "78af1771-9443-4d67-a451-e58862d2ef22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:32:05.381161 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.381119    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78af1771-9443-4d67-a451-e58862d2ef22-istio-podinfo" pod "78af1771-9443-4d67-a451-e58862d2ef22" (UID: "78af1771-9443-4d67-a451-e58862d2ef22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:32:05.381161 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.381115    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78af1771-9443-4d67-a451-e58862d2ef22-kube-api-access-qndbh" pod "78af1771-9443-4d67-a451-e58862d2ef22" (UID: "78af1771-9443-4d67-a451-e58862d2ef22"). InnerVolumeSpecName "kube-api-access-qndbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:32:05.479967 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.479889    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78af1771-9443-4d67-a451-e58862d2ef22-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:05.479967 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.479923    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78af1771-9443-4d67-a451-e58862d2ef22-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:05.479967 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.479939    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78af1771-9443-4d67-a451-e58862d2ef22-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:05.479967 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.479954    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qndbh\" (UniqueName: \"kubernetes.io/projected/78af1771-9443-4d67-a451-e58862d2ef22-kube-api-access-qndbh\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:32:05.558289 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:05.558260    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"]
Oct 30 20:32:05.561020 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:32:05.560995    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b56c575_497e_4524_98cc_9a0769ebc285.slice/crio-3cefa8d527bcfc7540fbaa73cecf0f5c155d452533612b4cae6b1acfe1e21c22 WatchSource:0}: Error finding container 3cefa8d527bcfc7540fbaa73cecf0f5c155d452533612b4cae6b1acfe1e21c22: Status 404 returned error can't find the container with id 3cefa8d527bcfc7540fbaa73cecf0f5c155d452533612b4cae6b1acfe1e21c22
Oct 30 20:32:06.182718 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:06.182690    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"
Oct 30 20:32:06.182964 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:06.182730    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2" event={"ID":"78af1771-9443-4d67-a451-e58862d2ef22","Type":"ContainerDied","Data":"457eccfdf2fe3a524054a3c1ec3181ee044845c43d68f7428128f66fb2d33b79"}
Oct 30 20:32:06.182964 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:06.182793    4823 scope.go:122] "RemoveContainer" containerID="7495441fb837316f1975bca972a32cb4e933d17499cc31fdae82543299e5acca"
Oct 30 20:32:06.184569 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:06.184538    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz" event={"ID":"5b56c575-497e-4524-98cc-9a0769ebc285","Type":"ContainerStarted","Data":"16ffb4e7d49c0925bd47c5e650c8f0530b8b34233c729a34bee57671f0cc4154"}
Oct 30 20:32:06.184679 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:06.184583    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz" event={"ID":"5b56c575-497e-4524-98cc-9a0769ebc285","Type":"ContainerStarted","Data":"3cefa8d527bcfc7540fbaa73cecf0f5c155d452533612b4cae6b1acfe1e21c22"}
Oct 30 20:32:06.204475 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:06.204415    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz" podStartSLOduration=2.204397987 podStartE2EDuration="2.204397987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:06.203887378 +0000 UTC m=+3726.096501701" watchObservedRunningTime="2026-10-30 20:32:06.204397987 +0000 UTC m=+3726.097012312"
Oct 30 20:32:06.217703 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:06.217673    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"]
Oct 30 20:32:06.222391 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:06.222365    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-de61d2-openshift-default-78c76cc477vmwb2"]
Oct 30 20:32:06.278080 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:06.278045    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78af1771-9443-4d67-a451-e58862d2ef22" path="/var/lib/kubelet/pods/78af1771-9443-4d67-a451-e58862d2ef22/volumes"
Oct 30 20:32:06.278367 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:06.278354    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:06.714974 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:06.714936    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:32:15.273425 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:15.273341    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:15.285958 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:15.285931    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:16.284821 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:16.284788    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:32:23.633233 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:23.633197    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:32:32.013036 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:32:32.012996    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:32:38.703625 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:38.703589    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:32:48.970216 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:48.970180    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"]
Oct 30 20:32:49.031496 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:49.031462    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"]
Oct 30 20:32:49.037603 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:49.037578    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c4ea9e5-5qbsj"]
Oct 30 20:32:50.278327 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:50.278290    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3e6ac7-bf62-4074-8b84-2903a30744c3" path="/var/lib/kubelet/pods/cb3e6ac7-bf62-4074-8b84-2903a30744c3/volumes"
Oct 30 20:32:50.882709 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:50.882679    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:32:57.794056 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:32:57.794023    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:33:02.013179 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:33:02.013138    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:33:03.990466 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:03.990401    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz" podUID="5b56c575-497e-4524-98cc-9a0769ebc285" containerName="main" containerID="cri-o://16ffb4e7d49c0925bd47c5e650c8f0530b8b34233c729a34bee57671f0cc4154" gracePeriod=45
Oct 30 20:33:04.253830 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.253774    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:33:04.306102 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.306065    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-tmp-dir\") pod \"5b56c575-497e-4524-98cc-9a0769ebc285\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") "
Oct 30 20:33:04.306284 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.306169    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-dshm\") pod \"5b56c575-497e-4524-98cc-9a0769ebc285\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") "
Oct 30 20:33:04.306284 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.306239    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b56c575-497e-4524-98cc-9a0769ebc285-kube-api-access-fbrfm\" (UniqueName: \"kubernetes.io/projected/5b56c575-497e-4524-98cc-9a0769ebc285-kube-api-access-fbrfm\") pod \"5b56c575-497e-4524-98cc-9a0769ebc285\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") "
Oct 30 20:33:04.306413 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.306306    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b56c575-497e-4524-98cc-9a0769ebc285-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b56c575-497e-4524-98cc-9a0769ebc285-tls-certs\") pod \"5b56c575-497e-4524-98cc-9a0769ebc285\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") "
Oct 30 20:33:04.306413 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.306339    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-tmp-dir" pod "5b56c575-497e-4524-98cc-9a0769ebc285" (UID: "5b56c575-497e-4524-98cc-9a0769ebc285"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:33:04.306413 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.306363    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-model-cache\") pod \"5b56c575-497e-4524-98cc-9a0769ebc285\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") "
Oct 30 20:33:04.306413 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.306393    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-home\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-home\") pod \"5b56c575-497e-4524-98cc-9a0769ebc285\" (UID: \"5b56c575-497e-4524-98cc-9a0769ebc285\") "
Oct 30 20:33:04.306710 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.306668    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-model-cache" pod "5b56c575-497e-4524-98cc-9a0769ebc285" (UID: "5b56c575-497e-4524-98cc-9a0769ebc285"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:33:04.307304 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.306790    4823 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-model-cache\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:04.307304 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.306810    4823 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-tmp-dir\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:04.307304 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.306790    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-home" pod "5b56c575-497e-4524-98cc-9a0769ebc285" (UID: "5b56c575-497e-4524-98cc-9a0769ebc285"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:33:04.308642 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.308621    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b56c575-497e-4524-98cc-9a0769ebc285-kube-api-access-fbrfm" pod "5b56c575-497e-4524-98cc-9a0769ebc285" (UID: "5b56c575-497e-4524-98cc-9a0769ebc285"). InnerVolumeSpecName "kube-api-access-fbrfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:33:04.308909 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.308891    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b56c575-497e-4524-98cc-9a0769ebc285-tls-certs" pod "5b56c575-497e-4524-98cc-9a0769ebc285" (UID: "5b56c575-497e-4524-98cc-9a0769ebc285"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:04.309019 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.308855    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-dshm" pod "5b56c575-497e-4524-98cc-9a0769ebc285" (UID: "5b56c575-497e-4524-98cc-9a0769ebc285"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:33:04.407652 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.407615    4823 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-dshm\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:04.407652 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.407648    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbrfm\" (UniqueName: \"kubernetes.io/projected/5b56c575-497e-4524-98cc-9a0769ebc285-kube-api-access-fbrfm\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:04.407652 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.407659    4823 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b56c575-497e-4524-98cc-9a0769ebc285-tls-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:04.408019 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.407667    4823 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b56c575-497e-4524-98cc-9a0769ebc285-home\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:04.621991 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.621962    4823 scope.go:122] "RemoveContainer" containerID="de092405d0157b72456e067796879e7d3249af8ad507f34a0e5d336caef62249"
Oct 30 20:33:04.712904 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.712855    4823 generic.go:356] "Generic (PLEG): container finished" podID="5b56c575-497e-4524-98cc-9a0769ebc285" containerID="16ffb4e7d49c0925bd47c5e650c8f0530b8b34233c729a34bee57671f0cc4154" exitCode=0
Oct 30 20:33:04.713075 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.712908    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"
Oct 30 20:33:04.713075 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.712940    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz" event={"ID":"5b56c575-497e-4524-98cc-9a0769ebc285","Type":"ContainerDied","Data":"16ffb4e7d49c0925bd47c5e650c8f0530b8b34233c729a34bee57671f0cc4154"}
Oct 30 20:33:04.713075 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.712973    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz" event={"ID":"5b56c575-497e-4524-98cc-9a0769ebc285","Type":"ContainerDied","Data":"3cefa8d527bcfc7540fbaa73cecf0f5c155d452533612b4cae6b1acfe1e21c22"}
Oct 30 20:33:04.713075 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.712988    4823 scope.go:122] "RemoveContainer" containerID="16ffb4e7d49c0925bd47c5e650c8f0530b8b34233c729a34bee57671f0cc4154"
Oct 30 20:33:04.722528 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.722513    4823 scope.go:122] "RemoveContainer" containerID="16ffb4e7d49c0925bd47c5e650c8f0530b8b34233c729a34bee57671f0cc4154"
Oct 30 20:33:04.722781 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:33:04.722761    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ffb4e7d49c0925bd47c5e650c8f0530b8b34233c729a34bee57671f0cc4154\": container with ID starting with 16ffb4e7d49c0925bd47c5e650c8f0530b8b34233c729a34bee57671f0cc4154 not found: ID does not exist" containerID="16ffb4e7d49c0925bd47c5e650c8f0530b8b34233c729a34bee57671f0cc4154"
Oct 30 20:33:04.722835 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.722790    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ffb4e7d49c0925bd47c5e650c8f0530b8b34233c729a34bee57671f0cc4154"} err="failed to get container status \"16ffb4e7d49c0925bd47c5e650c8f0530b8b34233c729a34bee57671f0cc4154\": rpc error: code = NotFound desc = could not find container \"16ffb4e7d49c0925bd47c5e650c8f0530b8b34233c729a34bee57671f0cc4154\": container with ID starting with 16ffb4e7d49c0925bd47c5e650c8f0530b8b34233c729a34bee57671f0cc4154 not found: ID does not exist"
Oct 30 20:33:04.745742 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.745712    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"]
Oct 30 20:33:04.749028 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:04.749004    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-381a9150-kserve-7d7f45b985-69fhz"]
Oct 30 20:33:05.717234 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:05.717201    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:33:06.297887 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:06.297332    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b56c575-497e-4524-98cc-9a0769ebc285" path="/var/lib/kubelet/pods/5b56c575-497e-4524-98cc-9a0769ebc285/volumes"
Oct 30 20:33:10.455889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:10.450691    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:33:15.354726 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.354693    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"]
Oct 30 20:33:15.355206 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.355192    4823 cpu_manager.go:414] "containerMap: removing container" podUID="78af1771-9443-4d67-a451-e58862d2ef22" containerName="istio-proxy"
Oct 30 20:33:15.355257 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.355208    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78af1771-9443-4d67-a451-e58862d2ef22" containerName="istio-proxy"
Oct 30 20:33:15.355257 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.355228    4823 cpu_manager.go:414] "containerMap: removing container" podUID="5b56c575-497e-4524-98cc-9a0769ebc285" containerName="main"
Oct 30 20:33:15.355257 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.355233    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b56c575-497e-4524-98cc-9a0769ebc285" containerName="main"
Oct 30 20:33:15.355257 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.355253    4823 cpu_manager.go:414] "containerMap: removing container" podUID="78af1771-9443-4d67-a451-e58862d2ef22" containerName="istio-proxy"
Oct 30 20:33:15.355257 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.355259    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78af1771-9443-4d67-a451-e58862d2ef22" containerName="istio-proxy"
Oct 30 20:33:15.355449 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.355354    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="78af1771-9443-4d67-a451-e58862d2ef22" containerName="istio-proxy"
Oct 30 20:33:15.355449 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.355363    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b56c575-497e-4524-98cc-9a0769ebc285" containerName="main"
Oct 30 20:33:15.355449 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.355371    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="78af1771-9443-4d67-a451-e58862d2ef22" containerName="istio-proxy"
Oct 30 20:33:15.360153 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.360132    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"
Oct 30 20:33:15.363401 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.363372    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:33:15.363401 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.363380    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:33:15.364077 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.364056    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\""
Oct 30 20:33:15.364996 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.364975    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:33:15.367530 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.367510    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"]
Oct 30 20:33:15.457052 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.457013    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbbdl\" (UniqueName: \"kubernetes.io/projected/7474b405-e352-487a-be5f-0d420642006d-kube-api-access-gbbdl\") pod \"maas-api-revoke-sub-5dada5ef-tbr9z\" (UID: \"7474b405-e352-487a-be5f-0d420642006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"
Oct 30 20:33:15.457052 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.457056    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7474b405-e352-487a-be5f-0d420642006d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dada5ef-tbr9z\" (UID: \"7474b405-e352-487a-be5f-0d420642006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"
Oct 30 20:33:15.457274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.457163    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7474b405-e352-487a-be5f-0d420642006d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dada5ef-tbr9z\" (UID: \"7474b405-e352-487a-be5f-0d420642006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"
Oct 30 20:33:15.558305 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.558270    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7474b405-e352-487a-be5f-0d420642006d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dada5ef-tbr9z\" (UID: \"7474b405-e352-487a-be5f-0d420642006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"
Oct 30 20:33:15.558460 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.558356    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbbdl\" (UniqueName: \"kubernetes.io/projected/7474b405-e352-487a-be5f-0d420642006d-kube-api-access-gbbdl\") pod \"maas-api-revoke-sub-5dada5ef-tbr9z\" (UID: \"7474b405-e352-487a-be5f-0d420642006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"
Oct 30 20:33:15.558460 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.558381    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7474b405-e352-487a-be5f-0d420642006d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dada5ef-tbr9z\" (UID: \"7474b405-e352-487a-be5f-0d420642006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"
Oct 30 20:33:15.558913 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.558893    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7474b405-e352-487a-be5f-0d420642006d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dada5ef-tbr9z\" (UID: \"7474b405-e352-487a-be5f-0d420642006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"
Oct 30 20:33:15.560752 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.560719    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7474b405-e352-487a-be5f-0d420642006d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dada5ef-tbr9z\" (UID: \"7474b405-e352-487a-be5f-0d420642006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"
Oct 30 20:33:15.576171 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.576144    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbbdl\" (UniqueName: \"kubernetes.io/projected/7474b405-e352-487a-be5f-0d420642006d-kube-api-access-gbbdl\") pod \"maas-api-revoke-sub-5dada5ef-tbr9z\" (UID: \"7474b405-e352-487a-be5f-0d420642006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"
Oct 30 20:33:15.675955 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.675924    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"
Oct 30 20:33:15.953854 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:15.953825    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"]
Oct 30 20:33:15.956564 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:33:15.956528    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7474b405_e352_487a_be5f_0d420642006d.slice/crio-db3819e4e6b31eff174eb843f4f034126442ce82bdf8f5392c50542ed18dea04 WatchSource:0}: Error finding container db3819e4e6b31eff174eb843f4f034126442ce82bdf8f5392c50542ed18dea04: Status 404 returned error can't find the container with id db3819e4e6b31eff174eb843f4f034126442ce82bdf8f5392c50542ed18dea04
Oct 30 20:33:16.806487 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:16.806449    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z" event={"ID":"7474b405-e352-487a-be5f-0d420642006d","Type":"ContainerStarted","Data":"bf7804e4b264ae1dbd9e8e4a5a4f47bf1ac5ce86ccc445d86aff1485c1e914bf"}
Oct 30 20:33:16.806891 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:16.806509    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z" event={"ID":"7474b405-e352-487a-be5f-0d420642006d","Type":"ContainerStarted","Data":"db3819e4e6b31eff174eb843f4f034126442ce82bdf8f5392c50542ed18dea04"}
Oct 30 20:33:16.824965 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:16.824913    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z" podStartSLOduration=1.82489989 podStartE2EDuration="1.82489989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:16.822908519 +0000 UTC m=+3796.715522840" watchObservedRunningTime="2026-10-30 20:33:16.82489989 +0000 UTC m=+3796.717514213"
Oct 30 20:33:17.813987 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:17.813947    4823 generic.go:356] "Generic (PLEG): container finished" podID="7474b405-e352-487a-be5f-0d420642006d" containerID="bf7804e4b264ae1dbd9e8e4a5a4f47bf1ac5ce86ccc445d86aff1485c1e914bf" exitCode=0
Oct 30 20:33:17.814435 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:17.813990    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z" event={"ID":"7474b405-e352-487a-be5f-0d420642006d","Type":"ContainerDied","Data":"bf7804e4b264ae1dbd9e8e4a5a4f47bf1ac5ce86ccc445d86aff1485c1e914bf"}
Oct 30 20:33:18.953291 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:18.953265    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"
Oct 30 20:33:18.997838 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:18.997807    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7474b405-e352-487a-be5f-0d420642006d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7474b405-e352-487a-be5f-0d420642006d-maas-api-xks-ca\") pod \"7474b405-e352-487a-be5f-0d420642006d\" (UID: \"7474b405-e352-487a-be5f-0d420642006d\") "
Oct 30 20:33:18.997838 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:18.997847    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7474b405-e352-487a-be5f-0d420642006d-kube-api-access-gbbdl\" (UniqueName: \"kubernetes.io/projected/7474b405-e352-487a-be5f-0d420642006d-kube-api-access-gbbdl\") pod \"7474b405-e352-487a-be5f-0d420642006d\" (UID: \"7474b405-e352-487a-be5f-0d420642006d\") "
Oct 30 20:33:18.998123 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:18.997926    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7474b405-e352-487a-be5f-0d420642006d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7474b405-e352-487a-be5f-0d420642006d-maas-api-openshift-ca\") pod \"7474b405-e352-487a-be5f-0d420642006d\" (UID: \"7474b405-e352-487a-be5f-0d420642006d\") "
Oct 30 20:33:18.998442 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:18.998407    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7474b405-e352-487a-be5f-0d420642006d-maas-api-openshift-ca" pod "7474b405-e352-487a-be5f-0d420642006d" (UID: "7474b405-e352-487a-be5f-0d420642006d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:33:19.000031 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:19.000000    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7474b405-e352-487a-be5f-0d420642006d-maas-api-xks-ca" pod "7474b405-e352-487a-be5f-0d420642006d" (UID: "7474b405-e352-487a-be5f-0d420642006d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:19.000172 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:19.000154    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7474b405-e352-487a-be5f-0d420642006d-kube-api-access-gbbdl" pod "7474b405-e352-487a-be5f-0d420642006d" (UID: "7474b405-e352-487a-be5f-0d420642006d"). InnerVolumeSpecName "kube-api-access-gbbdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:33:19.099899 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:19.099777    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7474b405-e352-487a-be5f-0d420642006d-maas-api-xks-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:19.099899 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:19.099811    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbbdl\" (UniqueName: \"kubernetes.io/projected/7474b405-e352-487a-be5f-0d420642006d-kube-api-access-gbbdl\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:19.099899 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:19.099827    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7474b405-e352-487a-be5f-0d420642006d-maas-api-openshift-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:19.827094 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:19.827058    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z" event={"ID":"7474b405-e352-487a-be5f-0d420642006d","Type":"ContainerDied","Data":"db3819e4e6b31eff174eb843f4f034126442ce82bdf8f5392c50542ed18dea04"}
Oct 30 20:33:19.827277 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:19.827108    4823 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db3819e4e6b31eff174eb843f4f034126442ce82bdf8f5392c50542ed18dea04"
Oct 30 20:33:19.827277 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:19.827068    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"
Oct 30 20:33:26.281583 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:26.281549    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:33:27.676911 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.676874    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-859f666b94-zssrp"]
Oct 30 20:33:27.677387 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.677370    4823 cpu_manager.go:414] "containerMap: removing container" podUID="7474b405-e352-487a-be5f-0d420642006d" containerName="revoke-keys"
Oct 30 20:33:27.677434 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.677388    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7474b405-e352-487a-be5f-0d420642006d" containerName="revoke-keys"
Oct 30 20:33:27.677489 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.677479    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="7474b405-e352-487a-be5f-0d420642006d" containerName="revoke-keys"
Oct 30 20:33:27.681001 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.680984    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:27.692325 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.692298    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:33:27.692462 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.692357    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:33:27.692767 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.692752    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-859f666b94-zssrp"]
Oct 30 20:33:27.718882 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.718840    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 20:33:27.719020 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.718921    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47lpr\""
Oct 30 20:33:27.719020 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.718926    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 20:33:27.792436 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.792409    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg48b\" (UniqueName: \"kubernetes.io/projected/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-kube-api-access-rg48b\") pod \"maas-api-859f666b94-zssrp\" (UID: \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:27.792577 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.792454    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-metrics-certs\") pod \"maas-api-859f666b94-zssrp\" (UID: \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:27.792577 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.792523    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-maas-api-tls\") pod \"maas-api-859f666b94-zssrp\" (UID: \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:27.792577 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.792574    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-openshift-service-ca\") pod \"maas-api-859f666b94-zssrp\" (UID: \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:27.893687 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.893650    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg48b\" (UniqueName: \"kubernetes.io/projected/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-kube-api-access-rg48b\") pod \"maas-api-859f666b94-zssrp\" (UID: \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:27.893879 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.893718    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-metrics-certs\") pod \"maas-api-859f666b94-zssrp\" (UID: \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:27.893879 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.893758    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-maas-api-tls\") pod \"maas-api-859f666b94-zssrp\" (UID: \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:27.893879 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.893841    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-openshift-service-ca\") pod \"maas-api-859f666b94-zssrp\" (UID: \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:27.894622 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.894593    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-openshift-service-ca\") pod \"maas-api-859f666b94-zssrp\" (UID: \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:27.896437 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.896408    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-metrics-certs\") pod \"maas-api-859f666b94-zssrp\" (UID: \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:27.896544 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.896516    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-maas-api-tls\") pod \"maas-api-859f666b94-zssrp\" (UID: \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:27.905194 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.905172    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg48b\" (UniqueName: \"kubernetes.io/projected/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-kube-api-access-rg48b\") pod \"maas-api-859f666b94-zssrp\" (UID: \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\") " pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:27.994576 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:27.994487    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:28.278695 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:33:28.278661    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f20d9e8_a7b4_44a2_b4e8_fdf2e9fee4fe.slice/crio-531da0fb4084e5edbe8e2d790cc483fbf4c3042effcded27953ea2321528b514 WatchSource:0}: Error finding container 531da0fb4084e5edbe8e2d790cc483fbf4c3042effcded27953ea2321528b514: Status 404 returned error can't find the container with id 531da0fb4084e5edbe8e2d790cc483fbf4c3042effcded27953ea2321528b514
Oct 30 20:33:28.278955 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:28.278934    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-859f666b94-zssrp"]
Oct 30 20:33:28.888889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:28.886111    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp" event={"ID":"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe","Type":"ContainerStarted","Data":"780fa91dc3ace75b6334d80b4cb3342f135d174f5127b60a837c436af5a5b893"}
Oct 30 20:33:28.888889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:28.886228    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp" event={"ID":"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe","Type":"ContainerStarted","Data":"531da0fb4084e5edbe8e2d790cc483fbf4c3042effcded27953ea2321528b514"}
Oct 30 20:33:28.888889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:28.886387    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:28.911254 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:28.911201    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp" podStartSLOduration=1.425977322 podStartE2EDuration="1.911188084s" totalImagesPullingTime="485.210762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:28.908363322 +0000 UTC m=+3808.800977645" watchObservedRunningTime="2026-10-30 20:33:28.911188084 +0000 UTC m=+3808.803802406"
Oct 30 20:33:32.013361 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:33:32.013311    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:33:34.898305 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:34.898271    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:35.305571 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.305483    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"]
Oct 30 20:33:35.310515 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.310491    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.313875 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.313825    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-33631e52-openshift-default-dockercfg-ttf8l\""
Oct 30 20:33:35.336109 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.336080    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"]
Oct 30 20:33:35.472888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.472200    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-envoy\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.472888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.472250    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-workload-certs\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.472888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.472302    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-workload-socket\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.472888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.472336    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-podinfo\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.472888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.472390    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-token\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.472888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.472422    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istiod-ca-cert\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.472888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.472456    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-credential-socket\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.472888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.472481    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d95v\" (UniqueName: \"kubernetes.io/projected/fd11e17c-7d24-4a98-b6a8-6199b0535c47-kube-api-access-8d95v\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.472888 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.472581    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-data\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.505780 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.505741    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-859f666b94-zssrp"]
Oct 30 20:33:35.506011 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.505981    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp" podUID="0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe" containerName="maas-api" containerID="cri-o://780fa91dc3ace75b6334d80b4cb3342f135d174f5127b60a837c436af5a5b893" gracePeriod=30
Oct 30 20:33:35.552889 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.552847    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"]
Oct 30 20:33:35.559029 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.558960    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"
Oct 30 20:33:35.569830 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.569800    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"]
Oct 30 20:33:35.573845 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.573815    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-token\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.573971 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.573879    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istiod-ca-cert\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.573971 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.573917    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-credential-socket\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.573971 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.573940    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d95v\" (UniqueName: \"kubernetes.io/projected/fd11e17c-7d24-4a98-b6a8-6199b0535c47-kube-api-access-8d95v\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.574147 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.574024    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-data\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.574147 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.574119    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-envoy\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.574273 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.574152    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-workload-certs\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.574273 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.574199    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-workload-socket\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.574273 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.574230    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-podinfo\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.575163 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.575132    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-credential-socket\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.575261 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.575163    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-workload-certs\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.575430 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.575394    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-workload-socket\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.575430 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.575418    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-data\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.575506 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.575463    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istiod-ca-cert\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.577169 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.577141    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-envoy\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.577664 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.577646    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-podinfo\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.588922 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.588895    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-token\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.589548 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.589525    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d95v\" (UniqueName: \"kubernetes.io/projected/fd11e17c-7d24-4a98-b6a8-6199b0535c47-kube-api-access-8d95v\") pod \"e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") " pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.624453 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.624360    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:35.675740 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.675325    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f-openshift-service-ca\") pod \"maas-api-68445fb9db-kcfzw\" (UID: \"3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"
Oct 30 20:33:35.675740 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.675380    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f-maas-api-tls\") pod \"maas-api-68445fb9db-kcfzw\" (UID: \"3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"
Oct 30 20:33:35.675740 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.675456    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f-metrics-certs\") pod \"maas-api-68445fb9db-kcfzw\" (UID: \"3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"
Oct 30 20:33:35.675740 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.675533    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjx88\" (UniqueName: \"kubernetes.io/projected/3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f-kube-api-access-kjx88\") pod \"maas-api-68445fb9db-kcfzw\" (UID: \"3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"
Oct 30 20:33:35.776379 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.776338    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f-openshift-service-ca\") pod \"maas-api-68445fb9db-kcfzw\" (UID: \"3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"
Oct 30 20:33:35.776379 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.776384    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f-maas-api-tls\") pod \"maas-api-68445fb9db-kcfzw\" (UID: \"3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"
Oct 30 20:33:35.776634 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.776517    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f-metrics-certs\") pod \"maas-api-68445fb9db-kcfzw\" (UID: \"3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"
Oct 30 20:33:35.776634 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.776623    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjx88\" (UniqueName: \"kubernetes.io/projected/3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f-kube-api-access-kjx88\") pod \"maas-api-68445fb9db-kcfzw\" (UID: \"3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"
Oct 30 20:33:35.777113 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.777085    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f-openshift-service-ca\") pod \"maas-api-68445fb9db-kcfzw\" (UID: \"3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"
Oct 30 20:33:35.779119 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.779097    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f-maas-api-tls\") pod \"maas-api-68445fb9db-kcfzw\" (UID: \"3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"
Oct 30 20:33:35.779241 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.779167    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f-metrics-certs\") pod \"maas-api-68445fb9db-kcfzw\" (UID: \"3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"
Oct 30 20:33:35.788142 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.788122    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjx88\" (UniqueName: \"kubernetes.io/projected/3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f-kube-api-access-kjx88\") pod \"maas-api-68445fb9db-kcfzw\" (UID: \"3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"
Oct 30 20:33:35.793378 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.793359    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:35.909720 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.909696    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"]
Oct 30 20:33:35.911785 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:33:35.911758    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd11e17c_7d24_4a98_b6a8_6199b0535c47.slice/crio-7dd539efd591f66383318f6dab9b46efae79eb2f57d88aec73e113e911885845 WatchSource:0}: Error finding container 7dd539efd591f66383318f6dab9b46efae79eb2f57d88aec73e113e911885845: Status 404 returned error can't find the container with id 7dd539efd591f66383318f6dab9b46efae79eb2f57d88aec73e113e911885845
Oct 30 20:33:35.913972 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.913940    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:33:35.914071 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.914005    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:33:35.914071 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.914035    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:33:35.919314 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.919293    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"
Oct 30 20:33:35.925006 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.924973    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn" event={"ID":"fd11e17c-7d24-4a98-b6a8-6199b0535c47","Type":"ContainerStarted","Data":"7dd539efd591f66383318f6dab9b46efae79eb2f57d88aec73e113e911885845"}
Oct 30 20:33:35.926856 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.926825    4823 generic.go:356] "Generic (PLEG): container finished" podID="0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe" containerID="780fa91dc3ace75b6334d80b4cb3342f135d174f5127b60a837c436af5a5b893" exitCode=0
Oct 30 20:33:35.927026 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.926966    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp" event={"ID":"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe","Type":"ContainerDied","Data":"780fa91dc3ace75b6334d80b4cb3342f135d174f5127b60a837c436af5a5b893"}
Oct 30 20:33:35.927026 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.927011    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp" event={"ID":"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe","Type":"ContainerDied","Data":"531da0fb4084e5edbe8e2d790cc483fbf4c3042effcded27953ea2321528b514"}
Oct 30 20:33:35.927187 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.927042    4823 scope.go:122] "RemoveContainer" containerID="780fa91dc3ace75b6334d80b4cb3342f135d174f5127b60a837c436af5a5b893"
Oct 30 20:33:35.927264 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.927204    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-859f666b94-zssrp"
Oct 30 20:33:35.941092 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.941072    4823 scope.go:122] "RemoveContainer" containerID="780fa91dc3ace75b6334d80b4cb3342f135d174f5127b60a837c436af5a5b893"
Oct 30 20:33:35.941449 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:33:35.941406    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"780fa91dc3ace75b6334d80b4cb3342f135d174f5127b60a837c436af5a5b893\": container with ID starting with 780fa91dc3ace75b6334d80b4cb3342f135d174f5127b60a837c436af5a5b893 not found: ID does not exist" containerID="780fa91dc3ace75b6334d80b4cb3342f135d174f5127b60a837c436af5a5b893"
Oct 30 20:33:35.941506 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.941467    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780fa91dc3ace75b6334d80b4cb3342f135d174f5127b60a837c436af5a5b893"} err="failed to get container status \"780fa91dc3ace75b6334d80b4cb3342f135d174f5127b60a837c436af5a5b893\": rpc error: code = NotFound desc = could not find container \"780fa91dc3ace75b6334d80b4cb3342f135d174f5127b60a837c436af5a5b893\": container with ID starting with 780fa91dc3ace75b6334d80b4cb3342f135d174f5127b60a837c436af5a5b893 not found: ID does not exist"
Oct 30 20:33:35.979286 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.978548    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-metrics-certs\") pod \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\" (UID: \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\") "
Oct 30 20:33:35.979286 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.978591    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-maas-api-tls\") pod \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\" (UID: \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\") "
Oct 30 20:33:35.979286 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.978692    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-kube-api-access-rg48b\" (UniqueName: \"kubernetes.io/projected/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-kube-api-access-rg48b\") pod \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\" (UID: \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\") "
Oct 30 20:33:35.979286 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.978733    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-openshift-service-ca\") pod \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\" (UID: \"0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe\") "
Oct 30 20:33:35.980922 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.980827    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-openshift-service-ca" pod "0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe" (UID: "0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:33:35.983677 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.983647    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-maas-api-tls" pod "0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe" (UID: "0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:35.983960 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.983931    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-kube-api-access-rg48b" pod "0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe" (UID: "0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe"). InnerVolumeSpecName "kube-api-access-rg48b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:33:35.985906 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:35.985707    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-metrics-certs" pod "0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe" (UID: "0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:36.080702 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:36.080659    4823 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-metrics-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:36.080702 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:36.080703    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-maas-api-tls\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:36.080997 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:36.080721    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg48b\" (UniqueName: \"kubernetes.io/projected/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-kube-api-access-rg48b\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:36.080997 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:36.080735    4823 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe-openshift-service-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:36.211300 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:36.211267    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"]
Oct 30 20:33:36.214145 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:33:36.214110    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8ce3be_1a6f_49ef_bdf2_5b7a4680c47f.slice/crio-e1c96a4c6427e0a3ddcdfe388dffc2acb14eca31b8d2fced40369915c3aabb51 WatchSource:0}: Error finding container e1c96a4c6427e0a3ddcdfe388dffc2acb14eca31b8d2fced40369915c3aabb51: Status 404 returned error can't find the container with id e1c96a4c6427e0a3ddcdfe388dffc2acb14eca31b8d2fced40369915c3aabb51
Oct 30 20:33:36.254546 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:36.254515    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-859f666b94-zssrp"]
Oct 30 20:33:36.259536 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:36.259509    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-859f666b94-zssrp"]
Oct 30 20:33:36.278547 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:36.278518    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe" path="/var/lib/kubelet/pods/0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe/volumes"
Oct 30 20:33:36.934659 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:36.934622    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn" event={"ID":"fd11e17c-7d24-4a98-b6a8-6199b0535c47","Type":"ContainerStarted","Data":"31b521d110a30e98b2a72163b11d4e8c9684b61565dbeabd9b98b0881a1b13e7"}
Oct 30 20:33:36.937279 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:36.937255    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw" event={"ID":"3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f","Type":"ContainerStarted","Data":"063167505bcdb8d1773f6711aa249a5911b44391efcf6bff8531ebefb1137427"}
Oct 30 20:33:36.937385 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:36.937298    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw" event={"ID":"3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f","Type":"ContainerStarted","Data":"e1c96a4c6427e0a3ddcdfe388dffc2acb14eca31b8d2fced40369915c3aabb51"}
Oct 30 20:33:36.937450 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:36.937403    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"
Oct 30 20:33:36.963270 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:36.963224    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn" podStartSLOduration=1.9632108499999998 podStartE2EDuration="1.96321085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:36.960127043 +0000 UTC m=+3816.852741367" watchObservedRunningTime="2026-10-30 20:33:36.96321085 +0000 UTC m=+3816.855825174"
Oct 30 20:33:36.999574 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:36.999524    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw" podStartSLOduration=1.610711381 podStartE2EDuration="1.999508249s" totalImagesPullingTime="388.796868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:36.998357989 +0000 UTC m=+3816.890972314" watchObservedRunningTime="2026-10-30 20:33:36.999508249 +0000 UTC m=+3816.892122579"
Oct 30 20:33:37.624703 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:37.624671    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:37.629646 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:37.629617    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:37.943459 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:37.943381    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:37.944292 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:37.944273    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:42.032587 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:42.032510    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:33:42.949224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:42.949196    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68445fb9db-kcfzw"
Oct 30 20:33:52.037673 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:52.037569    4823 generic.go:356] "Generic (PLEG): container finished" podID="fd11e17c-7d24-4a98-b6a8-6199b0535c47" containerID="31b521d110a30e98b2a72163b11d4e8c9684b61565dbeabd9b98b0881a1b13e7" exitCode=0
Oct 30 20:33:52.037673 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:52.037642    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn" event={"ID":"fd11e17c-7d24-4a98-b6a8-6199b0535c47","Type":"ContainerDied","Data":"31b521d110a30e98b2a72163b11d4e8c9684b61565dbeabd9b98b0881a1b13e7"}
Oct 30 20:33:52.038266 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:52.038086    4823 scope.go:122] "RemoveContainer" containerID="31b521d110a30e98b2a72163b11d4e8c9684b61565dbeabd9b98b0881a1b13e7"
Oct 30 20:33:52.039332 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:52.039306    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:33:52.039403 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:52.039371    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:33:52.039446 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:52.039401    4823 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 20:33:52.944127 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:52.944090    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:53.045541 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:53.045506    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn" event={"ID":"fd11e17c-7d24-4a98-b6a8-6199b0535c47","Type":"ContainerStarted","Data":"183bfab08139cadde2ddb0e91c109ec9468bf4a43dff25cf7494377229f21a6e"}
Oct 30 20:33:53.624705 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:53.624670    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:53.625897 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:53.625849    4823 patch_prober.go:29] interesting pod/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 30 20:33:53.625993 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:53.625925    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn" podUID="fd11e17c-7d24-4a98-b6a8-6199b0535c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 30 20:33:54.624717 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:54.624678    4823 patch_prober.go:29] interesting pod/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 30 20:33:54.625205 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:54.624743    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn" podUID="fd11e17c-7d24-4a98-b6a8-6199b0535c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 30 20:33:55.625694 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:55.625658    4823 patch_prober.go:29] interesting pod/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 30 20:33:55.626120 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:55.625710    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn" podUID="fd11e17c-7d24-4a98-b6a8-6199b0535c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 30 20:33:56.625396 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:56.625360    4823 patch_prober.go:29] interesting pod/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 30 20:33:56.625581 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:56.625418    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn" podUID="fd11e17c-7d24-4a98-b6a8-6199b0535c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 30 20:33:57.625124 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:57.625084    4823 patch_prober.go:29] interesting pod/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 30 20:33:57.625537 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:57.625177    4823 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn" podUID="fd11e17c-7d24-4a98-b6a8-6199b0535c47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 30 20:33:58.629489 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:58.629454    4823 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:58.629932 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:58.629895    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:33:58.630758 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:33:58.630739    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:34:02.013927 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:34:02.013890    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:34:04.652358 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:04.652325    4823 scope.go:122] "RemoveContainer" containerID="e7bf22b00287c47b6b5e2b32430e8e8cca2ef784559787d85c77ca3cd8d6de79"
Oct 30 20:34:04.663957 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:04.663929    4823 scope.go:122] "RemoveContainer" containerID="87bdcabb1d9983a2a60849d3c837854f4901d1239f8c68b9fccad3380ccf8b16"
Oct 30 20:34:07.435218 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:07.435181    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:34:11.016573 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:11.016538    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"]
Oct 30 20:34:11.017055 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:11.016767    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn" podUID="fd11e17c-7d24-4a98-b6a8-6199b0535c47" containerName="istio-proxy" containerID="cri-o://183bfab08139cadde2ddb0e91c109ec9468bf4a43dff25cf7494377229f21a6e" gracePeriod=30
Oct 30 20:34:13.632407 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:13.632368    4823 patch_prober.go:29] interesting pod/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:34:13.632814 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:13.632437    4823 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn" podUID="fd11e17c-7d24-4a98-b6a8-6199b0535c47" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:34:16.199086 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.199036    4823 generic.go:356] "Generic (PLEG): container finished" podID="fd11e17c-7d24-4a98-b6a8-6199b0535c47" containerID="183bfab08139cadde2ddb0e91c109ec9468bf4a43dff25cf7494377229f21a6e" exitCode=0
Oct 30 20:34:16.199467 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.199108    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn" event={"ID":"fd11e17c-7d24-4a98-b6a8-6199b0535c47","Type":"ContainerDied","Data":"183bfab08139cadde2ddb0e91c109ec9468bf4a43dff25cf7494377229f21a6e"}
Oct 30 20:34:16.199467 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.199218    4823 scope.go:122] "RemoveContainer" containerID="31b521d110a30e98b2a72163b11d4e8c9684b61565dbeabd9b98b0881a1b13e7"
Oct 30 20:34:16.279183 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.279126    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:34:16.433424 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.433391    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-envoy\") pod \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") "
Oct 30 20:34:16.433605 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.433448    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-data\") pod \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") "
Oct 30 20:34:16.433605 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.433480    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-credential-socket\") pod \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") "
Oct 30 20:34:16.433605 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.433520    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-workload-socket\") pod \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") "
Oct 30 20:34:16.433605 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.433564    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istiod-ca-cert\") pod \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") "
Oct 30 20:34:16.433929 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.433674    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-token\" (UniqueName: \"kubernetes.io/projected/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-token\") pod \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") "
Oct 30 20:34:16.433929 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.433701    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd11e17c-7d24-4a98-b6a8-6199b0535c47-kube-api-access-8d95v\" (UniqueName: \"kubernetes.io/projected/fd11e17c-7d24-4a98-b6a8-6199b0535c47-kube-api-access-8d95v\") pod \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") "
Oct 30 20:34:16.433929 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.433742    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-podinfo\") pod \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") "
Oct 30 20:34:16.433929 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.433731    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-data" pod "fd11e17c-7d24-4a98-b6a8-6199b0535c47" (UID: "fd11e17c-7d24-4a98-b6a8-6199b0535c47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:34:16.433929 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.433790    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-credential-socket" pod "fd11e17c-7d24-4a98-b6a8-6199b0535c47" (UID: "fd11e17c-7d24-4a98-b6a8-6199b0535c47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:34:16.433929 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.433810    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-workload-certs\") pod \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\" (UID: \"fd11e17c-7d24-4a98-b6a8-6199b0535c47\") "
Oct 30 20:34:16.434213 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.433947    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-workload-socket" pod "fd11e17c-7d24-4a98-b6a8-6199b0535c47" (UID: "fd11e17c-7d24-4a98-b6a8-6199b0535c47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:34:16.434213 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.434098    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istiod-ca-cert" pod "fd11e17c-7d24-4a98-b6a8-6199b0535c47" (UID: "fd11e17c-7d24-4a98-b6a8-6199b0535c47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:34:16.434213 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.434110    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-workload-certs" pod "fd11e17c-7d24-4a98-b6a8-6199b0535c47" (UID: "fd11e17c-7d24-4a98-b6a8-6199b0535c47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:34:16.434382 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.434351    4823 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-credential-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:16.434382 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.434379    4823 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-workload-socket\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:16.434590 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.434396    4823 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istiod-ca-cert\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:16.434590 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.434410    4823 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-workload-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:16.434590 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.434424    4823 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-data\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:16.435904 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.435871    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-envoy" pod "fd11e17c-7d24-4a98-b6a8-6199b0535c47" (UID: "fd11e17c-7d24-4a98-b6a8-6199b0535c47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:34:16.436000 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.435954    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd11e17c-7d24-4a98-b6a8-6199b0535c47-kube-api-access-8d95v" pod "fd11e17c-7d24-4a98-b6a8-6199b0535c47" (UID: "fd11e17c-7d24-4a98-b6a8-6199b0535c47"). InnerVolumeSpecName "kube-api-access-8d95v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:34:16.436000 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.435980    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-podinfo" pod "fd11e17c-7d24-4a98-b6a8-6199b0535c47" (UID: "fd11e17c-7d24-4a98-b6a8-6199b0535c47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:34:16.436349 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.436330    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-token" pod "fd11e17c-7d24-4a98-b6a8-6199b0535c47" (UID: "fd11e17c-7d24-4a98-b6a8-6199b0535c47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:34:16.535579 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.535503    4823 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-token\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:16.535579 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.535536    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d95v\" (UniqueName: \"kubernetes.io/projected/fd11e17c-7d24-4a98-b6a8-6199b0535c47-kube-api-access-8d95v\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:16.535579 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.535550    4823 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-podinfo\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:16.535579 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:16.535559    4823 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11e17c-7d24-4a98-b6a8-6199b0535c47-istio-envoy\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:17.206496 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:17.206467    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"
Oct 30 20:34:17.206496 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:17.206491    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn" event={"ID":"fd11e17c-7d24-4a98-b6a8-6199b0535c47","Type":"ContainerDied","Data":"7dd539efd591f66383318f6dab9b46efae79eb2f57d88aec73e113e911885845"}
Oct 30 20:34:17.207042 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:17.206543    4823 scope.go:122] "RemoveContainer" containerID="183bfab08139cadde2ddb0e91c109ec9468bf4a43dff25cf7494377229f21a6e"
Oct 30 20:34:17.277466 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:17.277440    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"]
Oct 30 20:34:17.305666 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:17.305638    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-33631e52-openshift-default-b9d6c7b45-4v7fn"]
Oct 30 20:34:18.030126 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.030097    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"]
Oct 30 20:34:18.030588 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.030576    4823 cpu_manager.go:414] "containerMap: removing container" podUID="0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe" containerName="maas-api"
Oct 30 20:34:18.030643 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.030589    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe" containerName="maas-api"
Oct 30 20:34:18.030643 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.030618    4823 cpu_manager.go:414] "containerMap: removing container" podUID="fd11e17c-7d24-4a98-b6a8-6199b0535c47" containerName="istio-proxy"
Oct 30 20:34:18.030643 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.030624    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd11e17c-7d24-4a98-b6a8-6199b0535c47" containerName="istio-proxy"
Oct 30 20:34:18.030643 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.030632    4823 cpu_manager.go:414] "containerMap: removing container" podUID="fd11e17c-7d24-4a98-b6a8-6199b0535c47" containerName="istio-proxy"
Oct 30 20:34:18.030643 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.030637    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd11e17c-7d24-4a98-b6a8-6199b0535c47" containerName="istio-proxy"
Oct 30 20:34:18.030821 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.030713    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd11e17c-7d24-4a98-b6a8-6199b0535c47" containerName="istio-proxy"
Oct 30 20:34:18.030821 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.030723    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd11e17c-7d24-4a98-b6a8-6199b0535c47" containerName="istio-proxy"
Oct 30 20:34:18.030821 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.030733    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f20d9e8-a7b4-44a2-b4e8-fdf2e9fee4fe" containerName="maas-api"
Oct 30 20:34:18.033682 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.033665    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:34:18.062616 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.062589    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-48059e-serving-cert\""
Oct 30 20:34:18.070029 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.068025    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"]
Oct 30 20:34:18.151796 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.151765    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080d5385-c633-4816-8157-2937249801a2-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7\" (UID: \"080d5385-c633-4816-8157-2937249801a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:34:18.151993 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.151813    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwsqv\" (UniqueName: \"kubernetes.io/projected/080d5385-c633-4816-8157-2937249801a2-kube-api-access-lwsqv\") pod \"maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7\" (UID: \"080d5385-c633-4816-8157-2937249801a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:34:18.151993 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.151912    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080d5385-c633-4816-8157-2937249801a2-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7\" (UID: \"080d5385-c633-4816-8157-2937249801a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:34:18.152080 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.152002    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080d5385-c633-4816-8157-2937249801a2-metrics-certs\") pod \"maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7\" (UID: \"080d5385-c633-4816-8157-2937249801a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:34:18.253403 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.253365    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080d5385-c633-4816-8157-2937249801a2-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7\" (UID: \"080d5385-c633-4816-8157-2937249801a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:34:18.253935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.253439    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwsqv\" (UniqueName: \"kubernetes.io/projected/080d5385-c633-4816-8157-2937249801a2-kube-api-access-lwsqv\") pod \"maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7\" (UID: \"080d5385-c633-4816-8157-2937249801a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:34:18.253935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.253485    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080d5385-c633-4816-8157-2937249801a2-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7\" (UID: \"080d5385-c633-4816-8157-2937249801a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:34:18.253935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.253545    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080d5385-c633-4816-8157-2937249801a2-metrics-certs\") pod \"maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7\" (UID: \"080d5385-c633-4816-8157-2937249801a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:34:18.254235 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.254205    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080d5385-c633-4816-8157-2937249801a2-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7\" (UID: \"080d5385-c633-4816-8157-2937249801a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:34:18.256031 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.256009    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080d5385-c633-4816-8157-2937249801a2-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7\" (UID: \"080d5385-c633-4816-8157-2937249801a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:34:18.256160 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.256052    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080d5385-c633-4816-8157-2937249801a2-metrics-certs\") pod \"maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7\" (UID: \"080d5385-c633-4816-8157-2937249801a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:34:18.265470 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.265441    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwsqv\" (UniqueName: \"kubernetes.io/projected/080d5385-c633-4816-8157-2937249801a2-kube-api-access-lwsqv\") pod \"maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7\" (UID: \"080d5385-c633-4816-8157-2937249801a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:34:18.280276 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.280217    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd11e17c-7d24-4a98-b6a8-6199b0535c47" path="/var/lib/kubelet/pods/fd11e17c-7d24-4a98-b6a8-6199b0535c47/volumes"
Oct 30 20:34:18.347410 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.347363    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:34:18.635008 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:18.634985    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"]
Oct 30 20:34:18.637688 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:34:18.637659    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080d5385_c633_4816_8157_2937249801a2.slice/crio-343c46cd3e24945861fcfde35a28b37d9861e742f797096d3876843c59dc9bf2 WatchSource:0}: Error finding container 343c46cd3e24945861fcfde35a28b37d9861e742f797096d3876843c59dc9bf2: Status 404 returned error can't find the container with id 343c46cd3e24945861fcfde35a28b37d9861e742f797096d3876843c59dc9bf2
Oct 30 20:34:19.220100 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:19.220068    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7" event={"ID":"080d5385-c633-4816-8157-2937249801a2","Type":"ContainerStarted","Data":"343c46cd3e24945861fcfde35a28b37d9861e742f797096d3876843c59dc9bf2"}
Oct 30 20:34:20.227851 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:20.227807    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7" event={"ID":"080d5385-c633-4816-8157-2937249801a2","Type":"ContainerStarted","Data":"2a03e706ccef1895a838de4c093543d3f1b7af7d60f3c539df9e1b5000a55643"}
Oct 30 20:34:20.228283 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:20.228015    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:34:20.248831 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:20.248746    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7" podStartSLOduration=1.712151658 podStartE2EDuration="2.248732701s" totalImagesPullingTime="536.581043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:20.247744132 +0000 UTC m=+3860.140358455" watchObservedRunningTime="2026-10-30 20:34:20.248732701 +0000 UTC m=+3860.141347025"
Oct 30 20:34:20.513670 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:20.513593    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:34:26.238535 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:34:26.238506    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:34:32.014480 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:34:32.014441    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:35:01.833022 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:35:01.832993    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:35:01.851522 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:35:01.851497    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:35:02.021732 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:35:02.021687    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:35:30.027908 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:35:30.027791    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"]
Oct 30 20:35:30.030511 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:35:30.030485    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-q6swp"]
Oct 30 20:35:30.277536 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:35:30.277499    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce424581-053b-4ee8-8fa6-81af5fa86e86" path="/var/lib/kubelet/pods/ce424581-053b-4ee8-8fa6-81af5fa86e86/volumes"
Oct 30 20:35:32.014159 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:35:32.014117    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:36:02.014760 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:36:02.014723    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:36:08.282338 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.282303    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"]
Oct 30 20:36:08.282921 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.282566    4823 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7" podUID="080d5385-c633-4816-8157-2937249801a2" containerName="maas-api" containerID="cri-o://2a03e706ccef1895a838de4c093543d3f1b7af7d60f3c539df9e1b5000a55643" gracePeriod=30
Oct 30 20:36:08.570905 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.570878    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:36:08.725117 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.725082    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/080d5385-c633-4816-8157-2937249801a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/080d5385-c633-4816-8157-2937249801a2-metrics-certs\") pod \"080d5385-c633-4816-8157-2937249801a2\" (UID: \"080d5385-c633-4816-8157-2937249801a2\") "
Oct 30 20:36:08.725309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.725162    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/080d5385-c633-4816-8157-2937249801a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080d5385-c633-4816-8157-2937249801a2-maas-api-tls\") pod \"080d5385-c633-4816-8157-2937249801a2\" (UID: \"080d5385-c633-4816-8157-2937249801a2\") "
Oct 30 20:36:08.725309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.725186    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080d5385-c633-4816-8157-2937249801a2-kube-api-access-lwsqv\" (UniqueName: \"kubernetes.io/projected/080d5385-c633-4816-8157-2937249801a2-kube-api-access-lwsqv\") pod \"080d5385-c633-4816-8157-2937249801a2\" (UID: \"080d5385-c633-4816-8157-2937249801a2\") "
Oct 30 20:36:08.725309 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.725270    4823 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080d5385-c633-4816-8157-2937249801a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080d5385-c633-4816-8157-2937249801a2-openshift-service-ca\") pod \"080d5385-c633-4816-8157-2937249801a2\" (UID: \"080d5385-c633-4816-8157-2937249801a2\") "
Oct 30 20:36:08.725709 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.725679    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080d5385-c633-4816-8157-2937249801a2-openshift-service-ca" pod "080d5385-c633-4816-8157-2937249801a2" (UID: "080d5385-c633-4816-8157-2937249801a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:08.727333 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.727305    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080d5385-c633-4816-8157-2937249801a2-maas-api-tls" pod "080d5385-c633-4816-8157-2937249801a2" (UID: "080d5385-c633-4816-8157-2937249801a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:08.727455 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.727397    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080d5385-c633-4816-8157-2937249801a2-metrics-certs" pod "080d5385-c633-4816-8157-2937249801a2" (UID: "080d5385-c633-4816-8157-2937249801a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:08.727455 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.727417    4823 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080d5385-c633-4816-8157-2937249801a2-kube-api-access-lwsqv" pod "080d5385-c633-4816-8157-2937249801a2" (UID: "080d5385-c633-4816-8157-2937249801a2"). InnerVolumeSpecName "kube-api-access-lwsqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:08.826737 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.826652    4823 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/080d5385-c633-4816-8157-2937249801a2-metrics-certs\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:08.826737 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.826681    4823 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/080d5385-c633-4816-8157-2937249801a2-maas-api-tls\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:08.826737 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.826693    4823 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwsqv\" (UniqueName: \"kubernetes.io/projected/080d5385-c633-4816-8157-2937249801a2-kube-api-access-lwsqv\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:08.826737 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.826709    4823 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/080d5385-c633-4816-8157-2937249801a2-openshift-service-ca\") on node \"ip-10-0-138-61.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:08.834665 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.834625    4823 generic.go:356] "Generic (PLEG): container finished" podID="080d5385-c633-4816-8157-2937249801a2" containerID="2a03e706ccef1895a838de4c093543d3f1b7af7d60f3c539df9e1b5000a55643" exitCode=0
Oct 30 20:36:08.834845 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.834686    4823 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"
Oct 30 20:36:08.834845 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.834710    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7" event={"ID":"080d5385-c633-4816-8157-2937249801a2","Type":"ContainerDied","Data":"2a03e706ccef1895a838de4c093543d3f1b7af7d60f3c539df9e1b5000a55643"}
Oct 30 20:36:08.834845 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.834802    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7" event={"ID":"080d5385-c633-4816-8157-2937249801a2","Type":"ContainerDied","Data":"343c46cd3e24945861fcfde35a28b37d9861e742f797096d3876843c59dc9bf2"}
Oct 30 20:36:08.834845 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.834839    4823 scope.go:122] "RemoveContainer" containerID="2a03e706ccef1895a838de4c093543d3f1b7af7d60f3c539df9e1b5000a55643"
Oct 30 20:36:08.845186 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.845169    4823 scope.go:122] "RemoveContainer" containerID="2a03e706ccef1895a838de4c093543d3f1b7af7d60f3c539df9e1b5000a55643"
Oct 30 20:36:08.845476 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:36:08.845458    4823 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a03e706ccef1895a838de4c093543d3f1b7af7d60f3c539df9e1b5000a55643\": container with ID starting with 2a03e706ccef1895a838de4c093543d3f1b7af7d60f3c539df9e1b5000a55643 not found: ID does not exist" containerID="2a03e706ccef1895a838de4c093543d3f1b7af7d60f3c539df9e1b5000a55643"
Oct 30 20:36:08.845520 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.845487    4823 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a03e706ccef1895a838de4c093543d3f1b7af7d60f3c539df9e1b5000a55643"} err="failed to get container status \"2a03e706ccef1895a838de4c093543d3f1b7af7d60f3c539df9e1b5000a55643\": rpc error: code = NotFound desc = could not find container \"2a03e706ccef1895a838de4c093543d3f1b7af7d60f3c539df9e1b5000a55643\": container with ID starting with 2a03e706ccef1895a838de4c093543d3f1b7af7d60f3c539df9e1b5000a55643 not found: ID does not exist"
Oct 30 20:36:08.858162 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.858117    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"]
Oct 30 20:36:08.862886 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:08.862845    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-48059e-844b95f5bf-mndc7"]
Oct 30 20:36:09.966310 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:09.966257    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:36:10.277661 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:10.277587    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080d5385-c633-4816-8157-2937249801a2" path="/var/lib/kubelet/pods/080d5385-c633-4816-8157-2937249801a2/volumes"
Oct 30 20:36:18.000102 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:18.000067    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:36:23.320835 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:23.320802    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:36:32.014719 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:36:32.014671    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:36:38.592124 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:36:38.592083    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:37:02.014716 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:37:02.014681    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:37:04.730053 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:04.730016    4823 scope.go:122] "RemoveContainer" containerID="f568cd048cf07927b5e640a777c3e23d8212a3ff2b45f46de5a176856f73d9b0"
Oct 30 20:37:04.739371 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:04.739344    4823 scope.go:122] "RemoveContainer" containerID="8027717e5a3b749e5892f81d410d3a4a2035a04bff209e13fc16db576f6b678a"
Oct 30 20:37:05.396041 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:05.396007    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gdgrn"]
Oct 30 20:37:16.667229 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:16.667197    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7b9fb75546-dcfw8_059a7cf4-196d-4ed8-b1d0-654d6e521c2a/manager/0.log"
Oct 30 20:37:16.880336 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:16.880302    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-kn9dm_f2e03817-be34-4666-befb-3a6cd283c45c/manager/0.log"
Oct 30 20:37:17.073302 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:17.073226    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-t5mbc_671e4edd-8d1a-4f5d-8375-1bba1686d004/manager/0.log"
Oct 30 20:37:17.827126 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:17.827102    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vql5z_beba841c-4b97-40ec-a959-f9130317dc4f/server/0.log"
Oct 30 20:37:18.007755 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:18.007719    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dxfcj_d945d1f6-e8e1-45fb-94e2-fe7d2751a813/manager/0.log"
Oct 30 20:37:18.999602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:18.999572    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-68445fb9db-kcfzw_3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f/maas-api/0.log"
Oct 30 20:37:19.925224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:19.925196    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5dada5ef-tbr9z_7474b405-e352-487a-be5f-0d420642006d/revoke-keys/0.log"
Oct 30 20:37:22.252851 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:22.252816    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-grls6_442a8c9e-848f-4a76-974c-b2d56c290340/manager/0.log"
Oct 30 20:37:22.668750 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:22.668722    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vkgjt_712ee475-078f-488e-aabb-35489b8bda90/kuadrant-console-plugin/0.log"
Oct 30 20:37:23.103487 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:23.103411    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gdgrn_1b499879-f7cb-4f24-95f4-083df601dd40/limitador/0.log"
Oct 30 20:37:23.303169 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:23.303138    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-fxhzx_4833c70b-80ee-494f-be73-0de577c66b69/manager/0.log"
Oct 30 20:37:24.301602 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:24.301570    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-669569fcd5ncl8n_911ae73c-fba8-4976-8c3c-38fb8ec9d20b/istio-proxy/0.log"
Oct 30 20:37:25.059201 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:25.059174    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-6kxj5_91e154c2-3f3b-49fb-ba0f-4c2be5d2388f/istio-proxy/0.log"
Oct 30 20:37:25.258677 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:25.258646    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-6p5n6_892f7495-a9e2-489d-92c5-12f35e1e65f3/istio-proxy/1.log"
Oct 30 20:37:25.834536 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:25.834506    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-cms2d_ff945b30-c4e9-4f06-b291-821104534ee7/istio-proxy/1.log"
Oct 30 20:37:26.561363 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:26.561337    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-rl8qk_f3c81047-57b2-4136-a7c7-cd930e059677/istio-proxy/0.log"
Oct 30 20:37:26.929435 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:26.929404    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k_f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac/istio-proxy/0.log"
Oct 30 20:37:27.109198 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:27.109164    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-p592q_d88a34ee-0b5d-4d35-b1bf-1679f82aab9f/payload-pre-processing/0.log"
Oct 30 20:37:27.494069 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:27.494039    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cb4fdcd5-zsxtw_df3b6a17-4bbf-41dc-83e5-df5172b87b7f/router/0.log"
Oct 30 20:37:27.942200 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:27.942173    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-6p5n6_892f7495-a9e2-489d-92c5-12f35e1e65f3/istio-proxy/0.log"
Oct 30 20:37:28.311122 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:28.311050    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-cms2d_ff945b30-c4e9-4f06-b291-821104534ee7/istio-proxy/0.log"
Oct 30 20:37:29.428155 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:29.428125    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs_7473cc37-e4fc-44cc-bb8b-4a42432af125/main/0.log"
Oct 30 20:37:30.173245 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:30.173215    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv_44597884-0bca-4006-a0cd-6bf9344678bc/main/0.log"
Oct 30 20:37:30.364791 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:30.364758    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-2wrpc_c2b23237-c311-4128-93c5-d6d8a3a53612/main/0.log"
Oct 30 20:37:30.723443 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:30.723410    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw_531e5023-637b-4a75-9846-8e53b962fc14/main/0.log"
Oct 30 20:37:32.014474 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:37:32.014435    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:37:41.019342 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:41.019311    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7b9fb75546-dcfw8_059a7cf4-196d-4ed8-b1d0-654d6e521c2a/manager/0.log"
Oct 30 20:37:41.219545 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:41.219518    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-kn9dm_f2e03817-be34-4666-befb-3a6cd283c45c/manager/0.log"
Oct 30 20:37:41.405949 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:41.405910    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-t5mbc_671e4edd-8d1a-4f5d-8375-1bba1686d004/manager/0.log"
Oct 30 20:37:42.050458 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:42.050428    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vql5z_beba841c-4b97-40ec-a959-f9130317dc4f/server/0.log"
Oct 30 20:37:42.231620 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:42.231584    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dxfcj_d945d1f6-e8e1-45fb-94e2-fe7d2751a813/manager/0.log"
Oct 30 20:37:42.977372 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:42.977345    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-68445fb9db-kcfzw_3c8ce3be-1a6f-49ef-bdf2-5b7a4680c47f/maas-api/0.log"
Oct 30 20:37:43.751779 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:43.751748    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5dada5ef-tbr9z_7474b405-e352-487a-be5f-0d420642006d/revoke-keys/0.log"
Oct 30 20:37:45.642204 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:45.642171    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-grls6_442a8c9e-848f-4a76-974c-b2d56c290340/manager/0.log"
Oct 30 20:37:46.029421 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:46.029336    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vkgjt_712ee475-078f-488e-aabb-35489b8bda90/kuadrant-console-plugin/0.log"
Oct 30 20:37:46.469437 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:46.469407    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gdgrn_1b499879-f7cb-4f24-95f4-083df601dd40/limitador/0.log"
Oct 30 20:37:46.660851 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:46.660819    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-fxhzx_4833c70b-80ee-494f-be73-0de577c66b69/manager/0.log"
Oct 30 20:37:47.273193 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:47.273160    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-669569fcd5ncl8n_911ae73c-fba8-4976-8c3c-38fb8ec9d20b/istio-proxy/0.log"
Oct 30 20:37:48.017874 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:48.017844    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-6kxj5_91e154c2-3f3b-49fb-ba0f-4c2be5d2388f/istio-proxy/0.log"
Oct 30 20:37:48.210208 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:48.210168    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-6p5n6_892f7495-a9e2-489d-92c5-12f35e1e65f3/istio-proxy/1.log"
Oct 30 20:37:48.790819 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:48.790787    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-cms2d_ff945b30-c4e9-4f06-b291-821104534ee7/istio-proxy/1.log"
Oct 30 20:37:49.549584 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:49.549555    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-rl8qk_f3c81047-57b2-4136-a7c7-cd930e059677/istio-proxy/0.log"
Oct 30 20:37:49.911555 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:49.911524    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k_f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac/istio-proxy/0.log"
Oct 30 20:37:50.090935 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:50.090892    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-p592q_d88a34ee-0b5d-4d35-b1bf-1679f82aab9f/payload-pre-processing/0.log"
Oct 30 20:37:50.444095 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:50.444068    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cb4fdcd5-zsxtw_df3b6a17-4bbf-41dc-83e5-df5172b87b7f/router/0.log"
Oct 30 20:37:50.989910 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:50.989855    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-wcqcs_7473cc37-e4fc-44cc-bb8b-4a42432af125/main/0.log"
Oct 30 20:37:51.723671 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:51.723626    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8bmvqv_44597884-0bca-4006-a0cd-6bf9344678bc/main/0.log"
Oct 30 20:37:51.903722 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:51.903694    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-2wrpc_c2b23237-c311-4128-93c5-d6d8a3a53612/main/0.log"
Oct 30 20:37:52.268712 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:37:52.268684    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-9z8vw_531e5023-637b-4a75-9846-8e53b962fc14/main/0.log"
Oct 30 20:38:02.014592 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:38:02.014552    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:38:19.079594 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:38:19.079513    4823 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"]
Oct 30 20:38:19.098062 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:38:19.098030    4823 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dada5ef-tbr9z"]
Oct 30 20:38:20.278996 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:38:20.278956    4823 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7474b405-e352-487a-be5f-0d420642006d" path="/var/lib/kubelet/pods/7474b405-e352-487a-be5f-0d420642006d/volumes"
Oct 30 20:38:32.014414 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:38:32.014369    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:39:02.014609 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:39:02.014571    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:39:23.523664 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:23.523634    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4zdqg_bbc7e365-75ff-4d4e-a0f9-88f22e5b8d8e/global-pull-secret-syncer/0.log"
Oct 30 20:39:23.728002 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:23.727971    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ntrs2_39126eba-223c-47c2-aff5-4233035260a3/connection-checker/0.log"
Oct 30 20:39:23.760701 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:23.760677    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-qjqz6_0af906a5-4f1e-4feb-8e1d-366cce9964c0/connection-checker/0.log"
Oct 30 20:39:23.797711 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:23.797637    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-v9m7c_28217977-99fe-4540-a322-3b3efb6dfa36/connection-checker/0.log"
Oct 30 20:39:23.971647 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:23.971617    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ncbz9_c246f08a-6fa3-4664-8a73-567d35f22d3a/konnectivity-agent/0.log"
Oct 30 20:39:24.018988 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:24.018958    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-61.us-east-2.compute.internal_f8f0f705095219e55fd87172f433b51c/haproxy/0.log"
Oct 30 20:39:30.127268 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:30.127232    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-grls6_442a8c9e-848f-4a76-974c-b2d56c290340/manager/0.log"
Oct 30 20:39:30.206050 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:30.205976    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vkgjt_712ee475-078f-488e-aabb-35489b8bda90/kuadrant-console-plugin/0.log"
Oct 30 20:39:30.307855 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:30.307822    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gdgrn_1b499879-f7cb-4f24-95f4-083df601dd40/limitador/0.log"
Oct 30 20:39:30.408784 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:30.408747    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-fxhzx_4833c70b-80ee-494f-be73-0de577c66b69/manager/0.log"
Oct 30 20:39:32.014230 ip-10-0-138-61 kubenswrapper[4823]: E1030 20:39:32.014194    4823 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8nh5p" containerName="extractor"
Oct 30 20:39:32.236203 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:32.236171    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-hlvg8_7427b4f7-fba8-4cde-ad90-1f46d5c92d87/cluster-monitoring-operator/0.log"
Oct 30 20:39:32.439613 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:32.439581    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j4tbd_fe1b3ffd-27d4-46fe-80e8-5920b5b62dac/node-exporter/0.log"
Oct 30 20:39:32.477610 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:32.477581    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j4tbd_fe1b3ffd-27d4-46fe-80e8-5920b5b62dac/kube-rbac-proxy/0.log"
Oct 30 20:39:32.530824 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:32.530798    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j4tbd_fe1b3ffd-27d4-46fe-80e8-5920b5b62dac/init-textfile/0.log"
Oct 30 20:39:33.002108 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.002069    4823 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"]
Oct 30 20:39:33.002588 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.002573    4823 cpu_manager.go:414] "containerMap: removing container" podUID="080d5385-c633-4816-8157-2937249801a2" containerName="maas-api"
Oct 30 20:39:33.002643 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.002588    4823 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080d5385-c633-4816-8157-2937249801a2" containerName="maas-api"
Oct 30 20:39:33.002709 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.002698    4823 memory_manager.go:365] "RemoveStaleState removing state" podUID="080d5385-c633-4816-8157-2937249801a2" containerName="maas-api"
Oct 30 20:39:33.006178 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.006157    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.012194 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.012162    4823 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rzbpr\"/\"default-dockercfg-czcz7\""
Oct 30 20:39:33.015364 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.015348    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rzbpr\"/\"openshift-service-ca.crt\""
Oct 30 20:39:33.015691 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.015385    4823 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rzbpr\"/\"kube-root-ca.crt\""
Oct 30 20:39:33.019973 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.019946    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"]
Oct 30 20:39:33.134404 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.134368    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a205b40-3669-4cf8-8467-31ec58f4ba4e-lib-modules\") pod \"perf-node-gather-daemonset-l9kk5\" (UID: \"3a205b40-3669-4cf8-8467-31ec58f4ba4e\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.134598 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.134429    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a205b40-3669-4cf8-8467-31ec58f4ba4e-podres\") pod \"perf-node-gather-daemonset-l9kk5\" (UID: \"3a205b40-3669-4cf8-8467-31ec58f4ba4e\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.134598 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.134452    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a205b40-3669-4cf8-8467-31ec58f4ba4e-proc\") pod \"perf-node-gather-daemonset-l9kk5\" (UID: \"3a205b40-3669-4cf8-8467-31ec58f4ba4e\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.134598 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.134540    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a205b40-3669-4cf8-8467-31ec58f4ba4e-sys\") pod \"perf-node-gather-daemonset-l9kk5\" (UID: \"3a205b40-3669-4cf8-8467-31ec58f4ba4e\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.134711 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.134600    4823 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj59h\" (UniqueName: \"kubernetes.io/projected/3a205b40-3669-4cf8-8467-31ec58f4ba4e-kube-api-access-cj59h\") pod \"perf-node-gather-daemonset-l9kk5\" (UID: \"3a205b40-3669-4cf8-8467-31ec58f4ba4e\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.236112 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.236065    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a205b40-3669-4cf8-8467-31ec58f4ba4e-lib-modules\") pod \"perf-node-gather-daemonset-l9kk5\" (UID: \"3a205b40-3669-4cf8-8467-31ec58f4ba4e\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.236316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.236148    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a205b40-3669-4cf8-8467-31ec58f4ba4e-podres\") pod \"perf-node-gather-daemonset-l9kk5\" (UID: \"3a205b40-3669-4cf8-8467-31ec58f4ba4e\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.236316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.236179    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a205b40-3669-4cf8-8467-31ec58f4ba4e-proc\") pod \"perf-node-gather-daemonset-l9kk5\" (UID: \"3a205b40-3669-4cf8-8467-31ec58f4ba4e\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.236316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.236205    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a205b40-3669-4cf8-8467-31ec58f4ba4e-sys\") pod \"perf-node-gather-daemonset-l9kk5\" (UID: \"3a205b40-3669-4cf8-8467-31ec58f4ba4e\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.236316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.236258    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a205b40-3669-4cf8-8467-31ec58f4ba4e-lib-modules\") pod \"perf-node-gather-daemonset-l9kk5\" (UID: \"3a205b40-3669-4cf8-8467-31ec58f4ba4e\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.236316 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.236287    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a205b40-3669-4cf8-8467-31ec58f4ba4e-proc\") pod \"perf-node-gather-daemonset-l9kk5\" (UID: \"3a205b40-3669-4cf8-8467-31ec58f4ba4e\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.236603 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.236315    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a205b40-3669-4cf8-8467-31ec58f4ba4e-podres\") pod \"perf-node-gather-daemonset-l9kk5\" (UID: \"3a205b40-3669-4cf8-8467-31ec58f4ba4e\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.236603 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.236258    4823 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj59h\" (UniqueName: \"kubernetes.io/projected/3a205b40-3669-4cf8-8467-31ec58f4ba4e-kube-api-access-cj59h\") pod \"perf-node-gather-daemonset-l9kk5\" (UID: \"3a205b40-3669-4cf8-8467-31ec58f4ba4e\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.236603 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.236321    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a205b40-3669-4cf8-8467-31ec58f4ba4e-sys\") pod \"perf-node-gather-daemonset-l9kk5\" (UID: \"3a205b40-3669-4cf8-8467-31ec58f4ba4e\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.248265 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.248240    4823 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj59h\" (UniqueName: \"kubernetes.io/projected/3a205b40-3669-4cf8-8467-31ec58f4ba4e-kube-api-access-cj59h\") pod \"perf-node-gather-daemonset-l9kk5\" (UID: \"3a205b40-3669-4cf8-8467-31ec58f4ba4e\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.319968 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.319903    4823 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:33.342032 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.342007    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-5s2v6_6945cea2-f314-48c5-b638-1b63a269a3f6/prometheus-operator-admission-webhook/0.log"
Oct 30 20:39:33.602252 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:33.602176    4823 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"]
Oct 30 20:39:33.605266 ip-10-0-138-61 kubenswrapper[4823]: W1030 20:39:33.605230    4823 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3a205b40_3669_4cf8_8467_31ec58f4ba4e.slice/crio-86dcc9c830ede5149ec00c3c268869646c37dbded328587fadae8ed01e0a821f WatchSource:0}: Error finding container 86dcc9c830ede5149ec00c3c268869646c37dbded328587fadae8ed01e0a821f: Status 404 returned error can't find the container with id 86dcc9c830ede5149ec00c3c268869646c37dbded328587fadae8ed01e0a821f
Oct 30 20:39:34.101560 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:34.101517    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5" event={"ID":"3a205b40-3669-4cf8-8467-31ec58f4ba4e","Type":"ContainerStarted","Data":"b54a89e276c8b9b6db14b4855141d008e3b99e8bcdaf48484d939fba879a3f9f"}
Oct 30 20:39:34.101998 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:34.101587    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:34.101998 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:34.101615    4823 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5" event={"ID":"3a205b40-3669-4cf8-8467-31ec58f4ba4e","Type":"ContainerStarted","Data":"86dcc9c830ede5149ec00c3c268869646c37dbded328587fadae8ed01e0a821f"}
Oct 30 20:39:34.119224 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:34.119177    4823 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5" podStartSLOduration=2.119165517 podStartE2EDuration="2.119165517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:34.118484362 +0000 UTC m=+4174.011098697" watchObservedRunningTime="2026-10-30 20:39:34.119165517 +0000 UTC m=+4174.011779840"
Oct 30 20:39:34.867070 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:34.867042    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-bwt58_0c796373-b082-4069-b5e7-228588cc99da/networking-console-plugin/0.log"
Oct 30 20:39:35.471667 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:35.471632    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/1.log"
Oct 30 20:39:35.480421 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:35.480397    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dp8wx_392e4212-393d-4aaa-904b-f72f1e287c34/console-operator/2.log"
Oct 30 20:39:36.182157 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:36.182115    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-xqs7x_2c48252d-5c0b-42fd-9e6c-c93d3e593379/download-server/0.log"
Oct 30 20:39:36.814810 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:36.814781    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-hv5j2_d4c7253d-c52b-4f85-bc06-cfafe6443bb4/volume-data-source-validator/0.log"
Oct 30 20:39:38.170286 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:38.170258    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kl9jq_ac07aa06-d891-4c95-8500-fdef3723ff38/dns/0.log"
Oct 30 20:39:38.203428 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:38.203386    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kl9jq_ac07aa06-d891-4c95-8500-fdef3723ff38/kube-rbac-proxy/0.log"
Oct 30 20:39:38.235029 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:38.234998    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2sdcp_b5f201ec-3c4d-4738-b721-ca4e445d63ad/dns-node-resolver/0.log"
Oct 30 20:39:39.068957 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:39.068921    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7cf748dbd6-ljbz6_899ea1d2-97ca-46c0-988a-886f70a01bed/registry/0.log"
Oct 30 20:39:39.178078 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:39.178038    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lcxk5_da310e19-8c58-4785-a88e-09edc8a63e2e/node-ca/0.log"
Oct 30 20:39:40.117811 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:40.117783    4823 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-l9kk5"
Oct 30 20:39:40.263857 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:40.263781    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-669569fcd5ncl8n_911ae73c-fba8-4976-8c3c-38fb8ec9d20b/istio-proxy/0.log"
Oct 30 20:39:43.448703 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:43.448674    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-6kxj5_91e154c2-3f3b-49fb-ba0f-4c2be5d2388f/istio-proxy/0.log"
Oct 30 20:39:43.511169 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:43.511138    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-6p5n6_892f7495-a9e2-489d-92c5-12f35e1e65f3/istio-proxy/0.log"
Oct 30 20:39:43.513927 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:43.513890    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-6p5n6_892f7495-a9e2-489d-92c5-12f35e1e65f3/istio-proxy/1.log"
Oct 30 20:39:43.635592 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:43.635562    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-cms2d_ff945b30-c4e9-4f06-b291-821104534ee7/istio-proxy/1.log"
Oct 30 20:39:43.635811 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:43.635792    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-cms2d_ff945b30-c4e9-4f06-b291-821104534ee7/istio-proxy/0.log"
Oct 30 20:39:43.788185 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:43.788112    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-rl8qk_f3c81047-57b2-4136-a7c7-cd930e059677/istio-proxy/0.log"
Oct 30 20:39:43.860703 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:43.860673    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k_f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac/istio-proxy/0.log"
Oct 30 20:39:43.893517 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:43.893483    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-p592q_d88a34ee-0b5d-4d35-b1bf-1679f82aab9f/payload-pre-processing/0.log"
Oct 30 20:39:43.971032 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:43.971004    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cb4fdcd5-zsxtw_df3b6a17-4bbf-41dc-83e5-df5172b87b7f/router/0.log"
Oct 30 20:39:44.102826 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:44.102743    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-669569fcd5ncl8n_911ae73c-fba8-4976-8c3c-38fb8ec9d20b/istio-proxy/0.log"
Oct 30 20:39:46.617044 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:46.617010    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vzpfb_3615ac3b-837c-46ea-b575-4cdc1edc78a0/serve-healthcheck-canary/0.log"
Oct 30 20:39:46.806847 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:46.806817    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-6kxj5_91e154c2-3f3b-49fb-ba0f-4c2be5d2388f/istio-proxy/0.log"
Oct 30 20:39:46.828851 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:46.828792    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-6p5n6_892f7495-a9e2-489d-92c5-12f35e1e65f3/istio-proxy/0.log"
Oct 30 20:39:46.831512 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:46.831489    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-6p5n6_892f7495-a9e2-489d-92c5-12f35e1e65f3/istio-proxy/1.log"
Oct 30 20:39:46.891983 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:46.891885    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-cms2d_ff945b30-c4e9-4f06-b291-821104534ee7/istio-proxy/1.log"
Oct 30 20:39:46.892187 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:46.892142    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-cms2d_ff945b30-c4e9-4f06-b291-821104534ee7/istio-proxy/0.log"
Oct 30 20:39:46.967743 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:46.967710    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-rl8qk_f3c81047-57b2-4136-a7c7-cd930e059677/istio-proxy/0.log"
Oct 30 20:39:47.003759 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:47.003734    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-749f59598c-sh98k_f5c3a3c0-a2ba-43ad-a083-ed7f6b0ff6ac/istio-proxy/0.log"
Oct 30 20:39:47.024274 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:47.024243    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-p592q_d88a34ee-0b5d-4d35-b1bf-1679f82aab9f/payload-pre-processing/0.log"
Oct 30 20:39:47.085938 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:47.085904    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cb4fdcd5-zsxtw_df3b6a17-4bbf-41dc-83e5-df5172b87b7f/router/0.log"
Oct 30 20:39:47.111722 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:47.111687    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-4vrds_6bc162c8-806a-4d1f-a046-3ade90069067/insights-operator/0.log"
Oct 30 20:39:47.114174 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:47.114154    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-4vrds_6bc162c8-806a-4d1f-a046-3ade90069067/insights-operator/1.log"
Oct 30 20:39:47.143812 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:47.143743    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8nh5p_b5ff1aae-17b6-4563-949b-e8321be54e25/kube-rbac-proxy/0.log"
Oct 30 20:39:47.175129 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:47.175103    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8nh5p_b5ff1aae-17b6-4563-949b-e8321be54e25/exporter/0.log"
Oct 30 20:39:47.207235 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:47.207208    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8nh5p_b5ff1aae-17b6-4563-949b-e8321be54e25/extractor/0.log"
Oct 30 20:39:47.481066 ip-10-0-138-61 kubenswrapper[4823]: I1030 20:39:47.480983    4823 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-pbql9-qfd8c_136b7cd5-518d-45d6-9458-ec837933526e/insights-gathering/0.log"